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CONTEXT OF THIS VOLUME 

This is one in a series of volumes produced by the JTPA EVALUATION DESIGN PROJECT. 



PURPOSE AND PHILOSOPHY 

The purpose of this project has been to develop a set of evaluation tools that are useful to states and local servke u 
areas (SDAs) .n judging the way their JTPA programs are being managed and the impact they are having The inl' 
nas been to base these analytic and managerial tools on sound program concepts and research methods and lo (k'- 
them such that the information obtained is of practical and direct use in improving JTPA policies and programs jf't 
state and local level This kind of information is also expected to make a unique contribution to national training po'i- 
and Federal oversight of JTPA. ■ ■ 

It is hoped that these volumes will stimulate and support state and local evaluation efforts in JTPA, and promote more 
consistency than in previous programs with respect to the issues studied and the methods used to investigate them An 
important goal is to encourage the generation of complementary information on program implementation and impaci 
that IS comparable across states and SDAs. Comprehensive, comparable information is essential to the development oi 
a valid and rehable knowledge base for resolving problems and improving programs. It is also required for adjusting na- 
tional training strategies to changing needs and priorities at the state and local level. 

PRODUCTS 

Consistent with this purpose and philosophy, the project has produced a set of materials to assist states and SDAs in 
evaluating their programs These are to be useful in planning, designing and implementing evaluation activities As an 
integrated collection, each set is developed to support comprehensive evaluations over the JTPA planning cycle. 

The careful tailoring of these materials to state and local users is appropriate. JTPA represents a new employment and 
traimng policy shaped not only by the experience of managers and the perspectives of employers, but by scientific assessmems 
ot previous approaches for addressing unemployment, poverty and other barriers to economic security. In this conte.Nt 
the value of JTPA programs is also expected to be judged. In fact, the Acfs assessment requirements are more explicit 
and sophisticated than those ol any employment and training legislation to date. It clearly distinguishes between moniior- 
ing activities, whose purpose is to determine compliance (such as with performance standards) and evaluation activities, 
wnose purpose is to determine how a program is being managed and implemented, and the kinds of effects it is havine 
L"c/«r!,'in,"c"/ relevant others. Equally significant, new constitutencies are e.xpected to make these more rigorous 
assessments. States and SDAs now have this important responsibility. It is the first time in the history of emoloyment 
and training programs that the Federal government's evaluation role has been significantly reduced. 

This change affords states and local areas opportunities to infiuence public policy. It also requires them to assume new 
oversight responsibilities. Program evaluation is expected to become an integral part of the management of organizations 
administering, planning and delivering public training services. This is as it should be. The more information available 
at these levels, where changes in organizations can most readily be made, the more effective the management of JTPA 
programs. This project was undertaken in that context. 

The evaluation tools produced by the project have been developed with a sensitivity to the differing needs, intere-^ts 
u Tf^LTr ^/^^""'^ ^^""'^ packaged into a single comprehensive and integrated set of volumes 

called JTPA Evaluation at the State and Local Level. The set contains planning and evaluation guides and issue papers 

The following volumes are available in the set: 



^"i*""^ _ _ Author ' 

I: Overview n • . -r- 

Project Team 

II: A General Planning Guide Deborah Feldman 

III: A Guide for Process Evaluations David Grembi vski 

III Supplement: Some Process Issues at the State Level David Grembowski 

IV: A Guide for Gross Impact Evaluations ^^^l sjmpson 

V: A Guide for Net Impact Evaluations j^^^y Johnson 

VI: An Implementation Manual for Net Impact Evaluations jerry Johnson 
VII: Issues Related to Net Impact Evaluations 

A. Issues in Evaluating Costs and Benefits Ernft Stromsdorfer 

B. The Debate Over Experimental vs. Quasi-Experimental Approaches Ann Blalock 
VIII: MIS Issues in Evaluating JTPA D^.j^ Grembowski 

NOTE: Although each of the discrete products listed above is the responsibility of a single author, each seeks to incor- 
pora e the results of professional peer review, the many excellent recommendations of the advisory group and 
and suggestions of the numerous practitioners interviewed in the process of developing these materials 



producive 1„ providing ".anagminSrS, oTcL^^^^^^^ «>PPl'. "^'^y 

In conclusion, several expectations have directed the development of these volumes: 

THE GUIDES 

The General Planning Guide 

oi.lnlf'fn J' '°. i" P'.^"'"^- ^""^'"^ dtve/op/n^ an organizational capacity to carry out process eross 

outcome, and net impact evaluations and to utilize their results. Separate state and local veSLS^Le availabfe ^ 

The Evaluation Gaides 
These volumes are to have the following characteristics: 

□The guides are to complement one another. 
•They are to provide information on program management and other characieristics of program implementation, which 

"impreValLrn? '"'^ °^ '"'^ "^''^'^ ^<imm.ms must be made in gross and net 

—Help explain the results of gross and net impact evaluations. 

'JypLTrc&:£ZT °" differentiated by type of service and 

"employeVS;^^^^^^^^ implementation modes and service strategies, and a broad array of client and 

—Help explain the results of net impact evaluations. 

-Suggest the more important outcomes that should be studied in net impact evaluations. 

-I-.elp sort out those aspects of implementation that may be most critical to study in process evaluations. 

•They are to provide information on net impact (the program's return on investment), which can: 
-Closely estimate the effect of the program's services on clients 

"wJf-I.^'^'f'' services and client groups are most important to study in broader but ie^s rigorous gross imoaci .ludie. 
iThe guides are to enable the user to carry out comprehensive assessments of JTPA programs 

*?h\Vp;Vi?.,Ta' p^o^r'S a°ni"wT;'"^ ''"^ ''''' °^ S^'" ^ -'^^ understanding of what 

IThe guides are to describe approaches and methodologies as consistently as possible, to achieve comparability. 
•They are to define variables and relationships as similarly as possible. 

•They are to define research designs, and methods of data collection and analysis using as similar concepts as possible. 

l^Z!ZnI^6^S!!: '° '"'^ ^"'^ ^^""'^d in Federal monitoring. and 

*I^itp1'° "J^*'.' '''l"'"' °/ 'he results of relevant CETA studies, national studies of JTPA. and issue oaoers 
on JTPA evaluation by national public interest organizations in the employment and training area ^ ^ 

•They are to rely on the professional literature in applied social research. 
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THE ISSUE PAPERS 



Volume VII contains two issue papers which serve as companion pieces to the preceding volumes on net impact evalua- 
tion. The first paper on cost-benefit issues is designed to help users identify, measure and analyze relationships between 
monetary and nonmonetary costs and benefits in determining the program's return on investment. The second paper ex- 
amines the pros and cons of different research strategies associated with the net impact approach. The final volume on 
MIS issues IS to assist users in better understanding how JTPA and other employment and training manasemeni informa^ 
tion systems can efficiently support the evaluation of program implementation and impact. 

THE SET OF VOLUMES 

The set \s integrated bui affords Hexible use. The user can utilize the entire set for comprehensive evaluations over 
a two-year plannmg cycle or longer planning period, or the user can apply the information in each volume independentlv, 
based on the most pressing evaluation priorities and timeframes and given the extent of resources, during a particular 
fiscal year or biennium. - ^ ^ 



It should be understood that although evaluation products have been developed for JTPA, their basic principles and 
methods can be applied more broadly by states and local areas to evaluate other employment and training programs and 
other social programs. & h ^ 

GENERAL ACKNOWLEDGMENTS 

The JTPA EVALUATION DESIGN PROJECT was developed and carried out based on the partnership philosophv 
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National Association of Counties: Kathy DuChene, Washington D.C. 
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DEFINITIONS OF TERMS 
USED IN THE GUIDE 



TERM 



Access decisions 
Allocative decisions 
Boundary 

Chain of objectives 
Consistency triangle 
Constraints (environmental) 
Conversion process 
Coordination component 
Creaming 

Decentralization 
Diagnosis 

Distributive outcomes 
Division of labor 
Dominance 

Employer satisfaction 

Environment 

Evaluation 

Feedback 

Final outcomes 

Function subcontracting 

Goals 

Governance 

Gross impact evaluation 
Gross outcomes 
Impact 

Impact evaluation 
Implementation strategy 
Inputs 
Interface 
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JTPA connection 


10 


vi I rM outcomes 


3 


Local autonomy 


25 


Management information system (MIS) 


28 


Market subcontracting 


50 


Mission component 


14 


Net impact evaluation 


4 


Net outcomes 


4 


Organ i zat i onal system 


7 


UU UpU US 


12 


rarticipant satisfaction 


97 


Performance control system 


28 


Personnel deci sions 


14 


Pigeonholing process 


68 


Planning process 


35 


riocess evaluation 


4 


rroper rit 


19 


Kevenue cecisions 


14 


Service creaming 


73 


Serv i ce i ntegr i ty 


72 


Service strategy 


7 


Service strength 


47 


Social component 


16 


Social consequences 


30 


ouaTT satisraction 


91 


Strategic planning 


15 


Theory of cause and effect 


5 


Treatment decisions 


15 


Validity assumptions 


60 


Work component 
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WHY SUBCONTRACT 



Previous studies of CETA show that subcontracting has twin benefits. [9] 
First, prime sponsors that subcontracted all services usually had better 
goal achievement than prime sponsors who provided all or part of the 
programs directly. Second, prime sponsors subcontracting all services 
also had higher placements and lower costs than prime sponsors providing 
all or some services, assuming competent subcontractors existed in the 
community. In short, the greater the level of subcontracting, the 
greater the performance (as measured in terms of goal achievement 
program costs and placements). 

These findings might be surprising to some people. We might have 
expected that providing most services directly to clients, where staff 
control is greatest, would result in better performance. Some of the 
following advantages of subcontracting suggest why it has worked so well 
under CETA: 

• Subcontracting encourages the efficient allocation of SDA 
resources. The SDA can choose where to put its funds among 
competing subcontractors. 

• The SDA can spread its risks across different agencies. In 
contrast, if one part fails in the service provider SDA, the 
entire SDA may perform poorly. 

• For SDAs providing services through satellite offices, this 
arrangement helps train managers. 

• The SDA can easily develop new programs (through the RFP 
process) or cancel programs that perform poorly. 

• Perhaps most importantly, subcontracting allows SDAs to 
concentrate on its mission component (goal setting, 
planning, monitoring performance), while subcontractors 
perfect service delivery operations, or the work component. 
In other words, this allows each organization to specialize, 
and with increased specialization comes the usual increases 
in effectiveness and efficiency. 

Of course, SDAs may have their own, practical reasons for subcontracting: 
a large number of competent agencies may exist in the community; the SDA 
may have inherited CETA staff, who administered contract programs; SDA 
staff lack skills for running training programs; or local agencies may 
put pressure on the SDA and local government to subcontract services. 
All of these f actor i influence SDA and PIC choices to consider 
subcontracting their services. 
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One fall, a small cricket found himself becoming colder and 
colder with each passing evening as the weather turned 
wintery. So he went to the wise old owl of the forest and 
said to him, "Oh wise old owl, please tell me what to do. 
The weather is getting colder and every evening I shiver and 
shak.> with the cold. If I don't do something soon, I know I 
will soon freeze to death. What can I do?" 

'|The answer to that is simple," said the wise old owl. 
Just turn yourself into a grasshopper, and hibernate for 
the winter." 

"But how can I turn myself into a grasshopper, oh wise old 
owl?" asked the cricket. 

Replied the owl, "Humph, don't bother me with the details. 
I ve given you the principle. You implement it." 

Mary Ann Scheirer 

PROGRAM IMPLEMENTATION 



With passage of the Job Training Partnership Act (JTPA) in 1982, Congress 
created new principles for organizing and operating the nation's largest 
employment and training program. States would assume a greater 
administrative role. Services to the economically disadvantaged would be 
provided through local "service delivery areas" (SDAs). New partnerships 
would be formed between the private sector and state and local govern- 
ments. Performance standards would be enforced. With the new 
principles, however, came very few details on implementing JTPA. No 
how-to books or other resources existed to guide implementation. 
States and SDAs soon realized that they were "on their own," and that 
successful implementation of JTPA would be a learning process as they 
ventured into new administrative territory. 

This is, in essence, what evaluation is, a learning process , that can be 
used to improve JTPA performance. Formally defined, evaluation is the 
collection and analysis of information by various methods to determine 
the effectiveness and efficiency of JTPA activities. In its simplest 
form, evaluation involves carefully examining JTPA to identify program 
elements that do and do not work, at what cost and for what reasons. In 
short, evaluation helps decision-makers pinpoint and understand the 
conditions associated with program success and failure. As such, it is a 
management tool. 

When these conditions become known, implementing the conditions necessary 
for effective and efficient service delivery becomes possible. If state 
and local evaluations of JTPA were conducted nationwide, they would 
likely indicate that substantial program variation exists across service 
delivery systems, making simple blanket endorsements ("it works") and 
wholesale rejections ("it doesn't work") less appropriate and meaningful. 

However, many states and SDAs are unprepared to conduct evaluations of 
JTPA programs. Prior to JTPA, few states and local areas had performed 
evaluations of their employment and training programs. If each agency 



independently developed its own evaluation design, much duplication of 
effort and inconsistency in the designs might result. To correct this 
situation, the National Commission for Employment Policy funded the 
Washington State Employment Security Department to develop evaluation 
designs for use at the state and local level. The designs are intended 
to provide guidance and some uniformity to JTPA evaluation efforts across 
states and local areas. [36] 



IMPACT EVALUATION 

In this project and most program evaluations, two types of evaluations 
may be ^ conducted, ^ impact and process. As its name implies, impact 
evaluations determine the results, or outcomes, produced by the program. 
Congress has defined three major outcomes for JTPA: increased employ- 
ment, increased earnings and reduced welfare dependency (Sec. 106). 
Congress also has mandated that a fourth outcome, retention in un- 
subsidized employment, be used in judging the success of the program. 

In an impact evaluation each outcome can be examined in three different 
ways: proximate, final and distributive. Proximate outcomes refer to 
the immediate results of program efforts. For example, proximate 
outcomes for a welding training program might include getting a job after 
training, or getting a job in welding or a related specialty. Final 
outcomes refer to the long-term results of the program— for example, 
whether an individual is still employed with increased earnings one year 
after training. Positive proximate outcomes do not show conclusively 
that a participant has improved his or her long-run earning capability. 
However, their presence does suggest that individuals are progressing 
toward stable employment, while their absence may indicate little 
progress toward longer-run benefits. 

The last type is distributive outcomes, which define proximate and final 
outcomes for specific target groups (e.g., welfare recipients, 
minorities, etc.) or geographic areas (e.g., rural vs. urban). 
Distributive outcomes reflect equity issues in service delivery (i.e., 
who receives the benefits of the program) and are a source of variation 
useful in explaining program outcomes. Congress has defined JTPA's two 
cin'ef distributive outcomes: among el igible appl icants , (1) services 
should be provided to those who can benefit from, and who are most in 
need of, such opportunities; and (2) services should be distributed 
equitably among substantial segments of the eligible population 
(Sec. 141). 

Outcomes may be measured at the completion of training (proximate), one 
year after training (final) or for distinct groups of participants, such 
as the handicapped (distributive.,* By themselves, these measures are 
commonly known as "gross outcomes or simpl e percentages , total s or 
averages describing JTPA performance. Most SDAs monitor gross outcomes 
of participants , but monitoring alone cannot identify which services 
produce the best outcomes. By performing a "gross impact evaluation," an 
SDA can determine the relative effectiveness of each service in achieving 
JTPA outcomes. 
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Gross outcomes, however, may be the result of a variety of factors, such 
as the services participants receive in JTPA, the education or job 
history of participants prior to entering JTPA, or changes in a local 
economy. The central aim of an impact evaluation is to differentiate 
among these alternative causes to isolate and measure those participant 
outcomes produced by JTPA alone. The latter measures are known as "net 
outcomes, and evaluations estimating these service effects on outcomes 
are known as "net impact evaluations." 

Thus, while gross outcomes merely document program results produced by 
any number and variety of factors, net outcomes pinpoint results caused 
by the program. Therefore, net outcomes are a more accurate measure of 
program achievements than gross outcomes. 



PROCESS EVALUATIONS 

Impact evaluations provide only part of the information needed in program 
evaluation. By their methods, outcome evaluations often treat programs 
as black boxes": what goes on inside the program (or box) to produce 
the results is rarely assessed. Thus, while impact evaluations inform us 
about the results produced by the progr-am, they rarely explain why the 
results were found. Process evaluations (also known as implementation 
assessments, or evaluations of program implementation) fill this know- 
ledge gap by analyzing the processes that produce program results. The 
common aim of most process evaluations is to describe the black box— to 
specify the interventions and implementation strategy that caused the 
program's outcomes. 

In a process evaluation, different questions are asked depending on 
program outcomes. In general, programs may have tha three possible 
outcomes listed on the left in Exhibit 1. These are translated, on the 
right, for JTPA. 





EXHIBIT 


1 




JTPA OUTCOMES 




OUTCOME 


JTPA 


I. 


Positive/beneficial effect 


Increased employment 
Increased earnings 
Reduced welfare dependency 


II. 


No effect 


No change in employment, 
earnings, or other outcomes 


III. 


Opposite, negative effect 
than that intended 


Increased welfare dependency 
Reduced earnings 
Reduced employment 
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For programs with positive outcomes, the key question in a process 
evaluation is "What worked?" [18,36] Little is gained from measuring the 
outcomes of a program that cannot be described. Knowing wh^ a program 
works can assure its continued success in the future, while also helping 
other SDAs duplicate the program in their own jurisdictions. 

For the remaining two outcomes in Exhibit 1 (no effect and negative 
effects), process evaluations can help decipher the reasons for in- 
adequate achievement. While somewhat simplified, programs fail to 
achieve their objectives for two reasons. [27] First, the program's 
services may not work as intended. In general, programs provide services 
to achieve a result. The typical reasoning is along the lines of, "If we 
do X now, then Y will result," where X is a service and Y is the intended 
outcome. These "if-then" relationships are known typically as a 
program's "theory of cause and effect." In JTPA, Congress has authorized 
a number of participant services (Sec. 204). Congress reasoned that if 
participants received these services, they would achieve increased 
employment and earnings and reduced welfare dependency. No iron-clad 
guarantee exists that JTPA's theory of cause and effect will work. If 
Congress^ theory is wrong, the services that participants receive may not 
produce intended results (as listed in Exhibit 1). 

A second reason why some programs fail is faulty implementation. This 
can happen in two ways. First, the wrong means of implementation may be 
chosen. That is, programs can fail simply because their implementation 
strategies do not work in the field. Second, a program with a sound 
implementation strategy may still fail if it is not implemented as 
intended. Issues such as these are a primary focus. 

Congress has defined the major elements of JTPA's implementation 
strategy. Some of these elements are listed below. (See Appendix A for 
a more detailed discussion of the^e elements). 

• Decentral ization (or transfer) of responsibi 1 ity from 
the federal level to the state and local levels. 

• A statewide service del ivery system composed of the 
state, SDAs, subcontractors and coordinating agencies, 
each having its own responsibilities in the system. 

• Public-private partnership: The State Job Training 
Coordinating Council and the Private Industry Counci 1 . 

• Performance standards. 

• Incentive and technical assistance funds. 

• Coordination criteria. 

• Eligibility standards. 

• The 40 percent youth allocation and the 70/30 percent 
adult allocation of Title II funds. 
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The list conveys Congress' thinking about how JTPA should work. Under 
the Reagan administration, JTPA was one of several programs where re- 
sponsibility was transferred from the federal level to states and local 
areas. Performance standards were established for two reasons: (1) to 
assure state and local accountability for public funds; and (2) to assure 
that JTPA's major outcomes—increased en^pioyment and earnings and 
reductions in welfare dependency— would be achieved. Mechanisms were 
also created to assure that JTPA's distributive outcomes would be 
achieved. Eligibility standards were imposed to limit entry to those 
most in need. Funds were allocated to specific groups, such as youth, 
adults and older workers. In addition, coordination plans and criteria 
were^ introduced to expand the scope of services available to JTPA 
participants. Incentive and technical assistance funds were also 
included to assure high levels of performance across SDAs. Clearly, many 
implementation decisions were left to states, SDAs, subcontractors and 
coordination agencies. However, these decisions are always made within 
the broader implementation strategy imposed by Congress. Like JTPA's 
services, no iron-clad guarantee exists that JTPA's implementation 
strategy will work as Congress intended. Furthermore, no guarantees 
exist that states, SDAs and subcontractors can implement JTPA as Congress 
intended. 

In short, JTPA is a giganitic, national experiment in employment and 
training policy. Congress created JTPA in response to serious unemploy- 
ment problems. JTPA's services are the experiment's treatment, or 
intervention; JTPA'i implementation strategy prescribes how these 
services should be provided. Will the experiment produce results 
intended by Congress? 

Programs are successful when good theory and good implementation combine 
to produce intended outcomes, as shown in cell 1 of Exhibit 2 below. [24] 
Programs with good implementation may still fail under a bad theory (cell 
number 2), just as bad implementation may destroy an otherwise 
theoretically sound intervention (cell number 3). When theory and 
implemenation problems occur separately, they may be corrected in some 
cases to produce positive outcomes. When both occur simultaneously, the 
choice is simple, discard the program (cell number 4). 

EXHIBIT 2 

REASONS FOR PROGRAM SUCCESS AND FAILURE 

THEORY OF CAUSE AND EFFECT 
("Service Strategy") 

GOOD BAD 



GOOD 1. No problem 2. Theory (policy) 

(program success) problem 

IMPLEMENTATION _^ 

STRATEGY 

BAD 3. Implementation 4. No problem 

(control) problem (discard 

program) 
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In an SDA process evaluation our first concern is defining an SDA*s 
service strategy (or theory of cause and effect) and implementation 
strategy in greater detail. A useful way of doing this is to construct 
models of program operations at the SDA level. A model of a program's 
service strategy defines the sequence of events in a program's inter- 
ventions, or treatments, and how they are linked to produce program 
results. A model of a program's implementation strategy specifies the 
organizational mechanisms for achieving results that work either to 
promote or inhibit program outcomes. (See Exhibit 1.) Together the two 
models focus the process evaluation on the program's critical features to 
form a framework for guiding later analyses. By further narrowing our 
focus on the manipulatable elements of these models, a process evaluation 
may conserve i^carce JTPA resources by providing information directly 
useful to SDA management for improving program outcomes. 

Chapter 1 presents a model of the SDA's service strategy and 
implementation strategy. The model views the SDA as an "organizational 
system." This simply means that the SDA is an organization composed of 
several parts that work together to produce JTPA outcomes. By 
identifying the parts and examining how they work, we can discover the 
reasons for program success or failure in an SDA. In short, the model 
helps structure the process evaluation, serving as a step-by-step guide 
for examining how an SDA operates, and how these operation's succeed—or 
fail--in achieving JTPA outcomes. 

After an overview of the SDA model is presented in Chapter 1, Chapters 2 
through 6 examine different parts of the model. Each chapter will 
describe a particular part, its relationship to other parts in the model, 
and methods for examining the part in your own organization. In a con- 
cluding chapter, methods are presented to help you interpret what you 
have found. By following these chapters, you will have performed a 
process evaluation of your SDA. 

This guide should be useful to administrators in SDAs and subcontractors. 
It is intended to be a management tool for solving day-to-day problems as 
well as for performing comprehensive reviews of SDA or subcontractor 
operations. Two caveats should be mentioned here. First, the handbook 
does iH>t prescribe solutions to specific problems. It does describe how 
you can study a problem, understand its causes, and formulate a workable 
solution that meets your specific situation. Second, methods for 
conducting process evaluations (of JTPA and other programs as well) are 
still in their infancy. The methods described in this guide are 
comprehensive and are based on current, state-of-the-art techniques 
specifically adapted to the JTPA program. Thus, the guide's methods 
represent just one of several possible, ways of performing a process 
evaluation. As SDAs gain experience with" process evaluations, the 
guide's methods will likely become more refined and sophisticated to help 
local administrators understand "what tmly goes on" in JTPA. 
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CHAPTER 1. OVERVIEW OF THE JTPA ORGANIZATIONAL SYSTEM 



The SDA Director had a problem with one of her 
subcontractors. The subcontractor was responsible for 
providing most of the SDA's youth services. During the 
first program year, the subcontractor had a good performance 
record. For the past several months, however, the sub- 
contractor had failed to meet its performance standards. 
Two months ago the SDA had increased its monitoring and 
compliance efforts, but this only seemed to make the problem 
worse. In fact, the subcontractor was now accusing the SDA 
as being the cause of the problem! Regardless of who was 
actually at fault, she thought the PIC would probably 
terminate the subcontractor's contract. But what if the SDA 
really was the source of the problem, as the subcontractor 
claimed? If this was true, similar performance problems 
might occur with other subcontractors in the future. She 
decided to perform a process evaluation to find the answers 
to her questions. The process evaluation begins with a 
model of "what goes on" in JTPA organizations at the local 
level. 



By approving the JTPA, Congress implemented a nationwide training 
program, ^ 

—to prepare youth and unskilled adults for entry into the 
labor force and to afford job training to those economically 
disadvantaged individuals and other individuals facing 
serious barriers to employment, who are in special need of 
such training to obtain productive employment. (Sec. 2) 

The emphasis here is on the participant, but, clearly, the success cf 
JTPA depends on the actions of at least three groups of people: 
participants, employers and JTPA administrators. In its simplest form, 
JTPA is a program for connecting participants with employers in local 
labor markets through authorized services specified in the Act, as shown 
in Exhibit 3 on the next page. 
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EXHIBIT 3: THE JTPA CONNECTION 
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In the next section we present a more detailed model of how SDAs and 
subcontractors operate to match employers and participants. Before we 
begin a word of caution is needed. Parts of this section have a air of 
Tinality as if the model perfectly matches what goes on in the JTPA 
service denvery system. That is not true, because all model s--whether 
they are of organizations, airplanes or persons-are a simplification of 
reality. Regl organizations are enormously complex. Models htlp us 
manage the complexity, to simplify it and sharpen its features, makina 
the organization easier to understand. 

The models presented in this section may not exactly match your own 
program or image of JTPA. But they should nonetheless serve as building 
blocks for developing and implementing process evaluations customized to 
your own program. 



MODEL OF THE JTPA ORGANIZATIONAL SYSTEM 

lln ^f.c^°-,i%^ ^■''^^^'^ ^2 3" "organizational system." 

Liu.ii.ib.l/J That IS. the SDA is just one organization in a larger 
environment that includes *he state, subcontractors and other service 
agencies, such as welfare. As an organization, an SDA is composed of 
several interrelated parts that work together to accomplish predetermined 
goals and objectives. A model of the SDA organizational system is 
presented in Exhibit 4 on the next page. The parts of the model and 
their interrelationships are known as the system's structure The flows 
of resources and information through the organization, shown by the 
arrows in Exhibit 4. are the system's processes . These flows are 
controlled by decision-making and key decisions are indicated in the 
model. Now. let's walk through the model, reviewing each part 
individually. ^ 

ENVIRONMENT. The flow of activity in the model begins with the 
environment, or everything outside the SDA organization. 2/ The 
environment includes other organizations in the service delivery area 
such as subcontractors and welfare agencies, federal and state laws and 
regulations affecting the SDA, and local economic, social and political 
conditions that influence service delivery. Because the environment 
normally consists of elements the SDA cannot control, the environment 
often represents fixed conditions, or constraints, to which the SDA must 



1/ Although we continually refer to the "SDA" throughout this 
discussion, the model can also be applied to subcontractors. 

2/ A BOUNDARY, the dashed line in Exhibit 4, separates the SDA 
organization from the environment. Everything outside the boundary 
is part of the environment; everything inside is part of the SDA 
organization. 
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EXHIBIT 4 
SDA PROCESS MODEL 



ENVIRONMENT 
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CONVERSION PROCESS 



IMPACT 
(Final Outcomes) 



COMPONENTS 1 


DECISIONS 
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Al locative 


Work 1 
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Coordination 1 




Social 1 





OUTPOTS 
Participants 
hired by 
Employers 
(Proximate 
Outcomes) 



Organization Boundary 
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adapt in order to survive in the long-run. 3/ For some fortunate SDAs, 
the environment can also help them achieve their goals and performance 
standards, such as by having a pool of well-qualified subcontractors to 
draw from. 

INPUTS. The environment is the source of inputs that the SDA uses to 
achieve its goals. Inputs consist of resources (such as participants, 
employers, staff, materials and supplies) and information (such as infor- 
mation about the local economy). The flow of inputs into the SDA is 
governed largely by revenue, personnel and access decisions. Revenue 
decisions govern the flow of money from the environment into the system. 
Although Congress determined formulas allocating funds among states and 
service delivery areas, the SDA must determine the entity receiving JTPA 
funds. SDAs can also decide whether to supplement JTPA funds with other 
local revenues. 

Personnel decisions govern the flow of specialized knowledge or technical 
expertise into the system. They determine who is hired and fired and 
whether consultants are used. They also determine PIC memberships within 
the requirements imposed by Congress (Sec. 102). These decisions are 
important because the quality of personnel often determine whether 
programs succeed or fail. 

Access decisions primarly govern the flow of participants and employers 
into^ the system. Congress has placed few restrictions on employer 
participation but has established eligibility requirements for parti- 
cipants. Among those eligible to receive services, access decisions 
determine who does and does not receive services under JTPA's limited 
funds. Access decisions have a great influence on SDA's distributive 
outcomes. 

OUTPUTS . Looking at the big picture (Exhibit 4), the SDA uses inputs to 
produce outputs— in this case, participants hired by employers (as well 
as participants with non-positive terminations). The outputs in 
Exhibit 4 correspond with the proximate outcomes discussed in the 
Introduction. 

CONVERSION PROCESS . Inputs are transformed into outputs through the 
conversion process, which consists of four components: mission, work, 
coordination and social. Each of these is described below. 

MISSION . The mission component defines the goals of the organization. 
Congress has established the purpose of JTPA in S.?ction 2 of the Act. 
Congress expanded this training goal by making SDA<i accountable for 
training outcomes through performance standards (Sec. 10b). However, in 
delegating accountability to SDAs, Congress also granted SDAs discretion 
to develop local goals tailored to address their local employment and 
training problems. 



3/ Technically speaking, an SDA cannot go bankrupt due to poor 
performance, but it can be radically reorganized by the Governor. 



This transfer of responsibility occurred because Congress assumed that 
local people were most knowledgeable of their employment and training 
problems, and that a more effective and efficient program (including 
increased coordination of services) would result by placing authority 
near the locus of the problem. [4, 5, 34] 

This means that goals will vary across SDAs. Each SDA develops its own 
set of goals in response to environmental constraints (e.g., local 
economic, social and political conditions), the values of PIC members and 
other considerations. Once goals are developed, specific objectives are 
usually assigned to each goal. The objectives serve as "indicators" that 
inform management whether a goal has been achieved. In principle, if an 
SDA achieves all of the objectives associated with a goal, then the SDA 
has achieved its goal. 

After the SDA has developed goals (and objectives), it must develop a 
strategy, or plan, for accompl ishig them. This activity is known 
commonly as strategic planning in most organizations. Several key 
decisions are made during this planning process. First, treatment 
decisions must be made about what services to offer participants, 
employers, and the community as a whole. (See Exhibit 4.) That is, the 
SDA decides the service strategies that will be used to increase 
employment, increase earnings and reduce welfare dependency among parti- 
cipants. (See the Introduction.) This includes decisions to enter into 
coordination agreements with outside agencies to expand the scope of 
services available to particpants. 

Second, access decisions , as defined earlier, are made to determine what 
target groups wfll receive the services. Third, al locative decisions are 
also made to determine how resources are allocated among the SDA's 
services and target groups. That is, allocative decisions determine the 
mix of money and personnel that will be applied to each participant and 
employer that the SDA serves. This includes deciding whether the SDA 
subcontracts all service delivery to other organizations, or whether the 
SDA delivers some or iall services itself. 

These decisions are documented in the SDA's job training plan, which can 
serve two purposes. First, the SDA can use the plan as a blueprint for 
guiding service delivery during implementation of the plan. Second, by 
submitting the plan to the state for approval, the SDA is complying with 
an administrative requirement for obtaining JTPA funds. The latter 
purpose can be the sole purpose of the plan in some SDAs. 

For SDAs that subcontract part or all of their services, the mission 
component also includes operation of the performance control system, 
which monitors subcontractor performance. This is discussed later in the 
Feedback section. 

Although the emphasis has been on the SDA, subcontractors are 
organizations, too, and have mission components as well. Thus, sub- 
contractors also perform their own goal-setting and strategic planning. 
This will be discussed in greater detail in Chapter 4. 
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In summary, the mission component defines the SDA's goals. Through 
fJ'^^^IR'"^ planning, a job training plan is developed that describes how 
the SDA will achieve its goals. Because of the planning flexibility 
allowed by Congress, goals vary across SDAs, likely causing parallel 
variations in proximate, final and distributive outcomes as systems are 
pushed in different directions. 

WORK. The work component is the organization's means of achieving its 
goals and includes the procurement of resources, such as participants 
(through outreach services), work flow procedures, the treatments 
participants receive, and the paths that participants and employers 
traverse through the system. The work component also includes non-JTPA 
services that participants can receive through coordination agreements, 
such as those with welfare agencies. The work component also contains 
SDA activities not related directly to service delivery, such as 
operating a local economic development program or conducting a study of 
the local economy. For a process evaluation, however, the key point is 
that the work component is where the SDA's service strategy is executed. 

Decision-making also occurs in the work component. For each participant 
(and, to a lesser extent, for each employer as well) staff decide who 
gets services ( access decisions) and what services these will be 
( al locative decisions ). 

If the SDA subcontracts all of its services, the work component is 
performed largely by other organizations. If the SDA subcontracts only a 
part of its services, the work component is divided among the SDA and 
subcontracting organizations. Under either service arrangement, service 
delivery and decision-making is distributed across several organizations. 
As a consequence, the coordination of services among providers becomes a 
central concern of the SDA. 

COORDINATION. Every organization of human activity creates two 
fundamental and opposing requirements: the division of labor into 
various tasks to be performed, and the coordination of these tasks to 
accomplish the activity. [16] The coordination component includes 
mechanisms for coordinating (1) SDA-provided services with subcontractor- 
provided services; (2) SDA and subcontractor-provided services with 
outside agency services (such as welfare); and (3) service delivery among 
subcontractors. Whether an SDA has all three coordination mechanisms 
depends on the treatment, allocative and access decisions made in the 
mission component. In addition, the coordination component also includes 
organizational responsibilities, communication patterns, and rule-making 
processes. 

SOCIAL . This component refers to the "social climate" of the 
organization: participant, employer and staff satisfaction and morale, 
the social norms of informal groups, participant, staff and employer 
needs, and the professional and mission-oriented values of the system. 
The social component has a special relationship with the previous three 
components of the conversion process. If an organization— SDA or 
otherwise— is to survive in the long-run, it must overcome four problems: 
(1) define its purpose (mission); (2) determine what means to use to 
achieve its mission in the environment (work); (3) coordinate its efforts 
(coordination); and (4) solve the above three problems with the minimum 
of strain and tension (social). [13] That is, goals are intended to 
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effect some change. Conflicts inevitably occur as resources are 
manipulated in the work and coordination components to achieve the 
organization's mission. Severe conflict can undermine an organization's 
effectiveness and eff iciency--if not threaten its very survival. 
Therefore, organizations seek to maintain tensions and conflicts at 
reasonable levels through various mechanisms, such as involving client/ 
employer/interest groups in the planning process, or through an "open- 
door" management style. Thus, just like tne work and coordination 
components , the soci al component al so pi ays an i mportant rol e in 
achieving the organization's goals. 

GOVERNANCE . The conversion of inputs into outputs is directed by 
governance. For an SDA, governance usually includes PIC members, the SDA 
director and managers. One primary function of governance is to 
establish the goals of the organization, and to make sure that its work, 
coordination and social components achieve them. 

IMPACT . Over the long run, SDA outputs have an impact on the 
environment. These impacts are reduced unemployment (or job retention), 
increased earnings and reduced welfare dependency among former 
participants, as measured by final outcomes. (See the Introduction.) 

FEEDBACK . Together, the output and impact elements of the model measure 
gross outcomes. Through the feedback process, both inform governance and 
management how well the system is operating. If they receive infor- 
mation, or feedback, indicating a discrepancy between system goals and 
output (such as a failure to meet performance standards), governance and 
management may alter the components of the conversion process to minimize 
or eliminate the difference. In most organizations, this feedback 
process is known as the performance control system, which is described in 
greater detail in Chapter 3. 

In summary, the model in Exhibit 4 describes organizational structures 
and processes common to most SDAs and subcontractors in JTPA. In terms 
of the SDA process evaluation^ everything inside the SDA's boundary (the 
dashed line in Exhibit 4) constitutes the SDA's implementation strategy . 
(See the Introduction.) The SDA's service strategy consists of services 
offered to participants and employers in the work component. Thus, the 
SDA's service strategy is a subpart of the agency's overall implement- 
ation strategy. 

Up until this point only the structure and processes of the 
organizational model have been presented. In the next section we 
complete the model with the two conditions that determine whether the 
organization operates well or poorly. 



CONDITIONS REQUIRED FOR ORGANIZATIONAL SUCCESS 

Normally, organizations operate well when two conditions are satisfied. 
First , all four components — mission , work, coordination and social—must 
be involved in the conversion process . Problems usually occur when one 
component has been overlooked. For example, an SDA may not have 
developed any goals and, as a consequence, effort is rarely directed at 
achieving some ultimate purpose. Similar examples for the other 
components are listed below. 
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COMPONENT 



EXAMPLE 



No goals exist. 

Participants receive 
little training. 

No mechani sms exi st for 
coordi nati ng servi ces 
among agencies. 

No mechanisms exist for 
solving staff grievances, 
causing low staff morale. 

Thus, one of the key tasks in a SDA process evaluation is to examine 
whether all four components are involved in the conversion process. 
However, caution must always be exercised in performing this task. 
Organizations sometimes emphasize one component over the others. Given 
that JTPA is a relatively new program, it may be appropriate for SDAs to 
concentrate on different problems— or components in the organization— at 
different stages of their development. 

It might be argued, for example, that at the beginning the 
first^ concern of (the SDA) is to identify tangible 
activities it can undertake, even if its mission is still 
rather ambiguous or ill defined. The SDA is expected to 
show action. It thus puts emphasis on (the work component), 
but soon thi s preoccupati on wi th doi ng thi ngs creates 
confusion. No one knows what anyone else is doing; 
duplication of efforts are brought to light; in short, the 
need is seen to shift to a primary emphasis on (co- 
ordi nati on) . As the (SDA) gets i ts i nternal management 
under control, questions begin to be asked about whether the 
(SDA) is^ really making a difference in the outside world by 
all of its w^ll-organized activity. Negative responses to 
these questions occasion a shift in problem emphasis to goal 
attainment (mission component). Interest sti 1 1 centers 
around the evaluation of activity, but evaluation now is in 
terms of what changes are occurring in society as the result 
of the organization's (or JTPA's) efforts. Finally, when it 
is decided that the (SDA and JTPA) are effecting the desired 
changes in society, primary concern shifts to the (social 
component). As this occurs, suggestions for change in 
methods or goals begin to be resisted on the grounds that 
such changes would interfere with the operation of a 
smoothly running, wel 1 -assembled system. Clients and 
legislators are consequently courted to help maintain the 
organization. [13: 65-66] 

So, at any given time i n its development, an SDA may place primary 
emphasis on one of the four components of its conversion process. Thus, 
while all four components must be addressed in the conversion process, 
emphasis among the four components may change as the SDA evolves, re- 
flecting the evolutionary development of the program. 
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The second 
fit." [16] That is, 
organization operates 



condition which an 



^ organization must satisfy is "proper 

organizations are composed of several parts. An 
well when it achieves a proper fit among the 
parts—making sure they are internally consistent, meshing together in 
synchrony to produce outcomes consistent with SDA goals. In terms of the 
organizational model in Exhibit 4, the chief parts of interest are the 
four components of the conversion process. SDAs and subcontractors 
operate well when they have achieved a proper fit (1) between the mission 
component and the environment , and (2) among the four components of the 
conversion process . Each of these requirements is described below. 

The^ first requirement for proper fit is that the environment and the 
SBA's mission component be consistent with each other. Reviewing 
material presented earlier, the environment contains constraints- 
elements that the SDA cannot control—to which the SDA must adapt in 
order to survive in the long run. Two major environmental constraints 
facing SDAs are the Act itself and local conditions, as shown in 
Exhibit 5 below^ 
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The Job Training Partnership Act is perhaps the major environmental 
constraint affecting aVI SDAs. Congress defined" the major goals, or 
outcomes, for JTPA, and established three important mechanisms- 
performance standards, eligibility requirements and target groups, as 
well as SDA resource allocation formulas-to achieve them. In principle 
these mechanisms should work if: >- r . 

• The performance standards actually contribute to the 
achievement of JTPA's goals (as defined by Congress). 

The eligibility requirements and target groups actually 
identify those most in need in the local area. 

• The resource allocation formulas in the Act provide 
sufficient funds to accomplish (1) and (2) above. 

In short. Congress (and to a lesser extent DDL and the states) have 
largely determined whom the SDA will serve, what the level of performance 
will be, and how much money each SDA will have to accomplish the JTPA's 
three major outcomes. If the three mechanisms are based on incorrect 
assumptions or are grossly inconsistent with each other, the Congres- 
sional mandated implementation strategy could fail and the Act would 
probably not achieve the results intended. 

These Congressional constraints, along with others imposed by local 
conditions, are revvjwed by Governance— usual ly the PIC. Based on its 
own values, interests and interpretations of these constraints, the PIC 
establishes a set of goals for the SDA to accomplish in the future. In 
principle, the SDA's goals and environmental constraints should be 
consistent with each other; that is, a proper fit should exist between 
the two. 

Whether an SDA's goals are consistent or inconsistent with its 
environmental constraints is determined by comparing the goals with the 
major constraints in Exhibit 5. For example, one SDA in an area of low 
unemployment achieves a proper fit by developing services oriented toward 
providing local employers with trained workers that meet their labor 
needs. Adequate resources and a sufficient supply of eligible applicants 
are available to carry out the SDA's mission. A proper fit between 
mission and environment exists. 

In contrast, an SDA serving an urban ghetto with severe social, economic 
and political problems may set "comprehensive services to participants" 
as its primary goal. To assure that only those most in need receive 
services, the PIC also adopts "being unemployed" as an additional 
eligibility requirement. Even so, the SDA is overwhelmed with applicants 
and has insufficient resources to achieve its performance standards. In 
some cases participants are trained for jobs that do not exist in the 
labor market. Here, a proper fit is not fully achieved. The mission 
component must be realigned to achieve a proper fit with its environment 
by making one or more of the following changes: changing eligibility 
requirements (to limit access to services); reducing its performance 
standards (so comprehensive services can be provided to eligible 
applicants); obtaining additional funds from other sources (such as 
incentive/technical assistance funds); or improving the link between 
training and labor needs of the market. Obviously, the SDA must seek 
state approval to perform some of these actions. 



The other proper fit requirement is that the four components of the 
conversion process be consistent with each other. Normally, goals are 
developed first in the mission component. Then the work and coordination 
components are designed to achieve the goals in ways that satisfy those 
involved in the conversion process (social component). A common cause of 
SDA (and subcontractor) problems is inconsistent mission, work and co- 
ordination components, which cause social friction and conflicts in the 
service delivery process. For example, the SDA may have goals (mission 
component) to treat the hard-to-serve but mainly offer placement services 
(work component), better suited for job-ready participants. Or the SDA 
may have developed coordination plans with local welfare offices (mission 
component)^ but in reality only a handful of welfare applicants are 
referred to the SDA (coordination component). 



HOW TO USE THE MODEL 

This chapter began with an SDA director experiencing a subcontractor 
problem. She decided to conduct a process evaluation to discover the 
causes of the problem. The central piece of the process evaluation is a 
model of the organization, either SDA or subcontractor. The model 
consists of parts that work together to produce outcomes. The parts of 
the model were described, and two conditions required for good operation 
were presented. Performing a process evaluation means applying the model 
and the two conditions in an SDA or subcontractor organization. That is, 
each part of the organization is examined to determine whether the two 
conditions are satisfied. The organization having a problem usually has 
failed to satisfy one or both of the conditions. 

The remaining chapters describe how you can perform a process evaluation 
in your own SDA. Each chapter is devoted to one part of model (in 
Exhibit 4). In each chapter, the part is described in greater detail, 
and guidelines are presented for examining your own organization. The 
guidelines are illustrated using the SDA problem presented on the first 
page of this chapter. Recall that the methods can be used either to 
solve specific SDA/subcontractor problems or to explain an SDA's success 
or failure in achieving specific gross outcomes. So, let's begin with 
the organization's environment in Chapter 2. 
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CHAPTER 2 
ENVIRONMENT, GOVERNANCE AND THE 
PERFORMANCE CONTROL SYSTEM 



Understanding an organization's environment is essential before we begin the 
process evaluation in Chapter 3. Here we look at the following elements: 

Characteristics of the Statewide Service Delivery System 

How Local Conditions MLy Affect SDA Operations and the 
Achievement of the Organization's Goals, Performance Standards, 
and JTPA Outcomes (Increased Employment and Earnings and 
Reduced Welfare Dependency) 

The Role of Governance in the Performance Control System 

Description of the SDA Performance Control System 

Use and Misuse of Performance Control Systems 

Economic and Social Consequences of Performance Control 

Systems 

Summary 
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CHAPTER 2. ENVIRONMENT, GOVERNANCE AND THE 
PERFORMANCE CONTROL SYSTEM 



An SDA process evaluation is a tool that SDA directors can use either to 
explain gross outcomes or to solve problems. Faced with the sub- 
contractor performance problem described on the first page of Chapter i , 
the SDA director's initial response is to begin a careful review of her 
organization's mission, work, coordination and social components. In 
simple organizations this is a proper course, but an SDA is a complex 
organization that operates in a complex environment. Understanding this 
environment can yield insights about the underlying causes of many 
problems confronting management. 

In this chapter we take a closer look at the service delivery environment 
in which SDAs exist. Then, we will examine the SDA's performance control 
system— the chief tool that management has for monitoring SDA performance 
in this environment. Both will increase the SDA director's understanding 
of xhe causes of the problem at hand, although discovering its exact 
cause(s) must await further examination in later chapters. 



THE SDA'S ENVIRONMENT: THE SERVICE DELIVERY SYSTEM 

Perhaps the most important factor in the SDA*s environment is the federal 
legislation authorizing JTPA. In Chapter 2 four major influences of the 
JTPA on SDAs are described: JTPA outcomes, 4/ SDA resource allocation 
formulas, performance standards and eligibility requirements. These are 
not the only federal influences on the SDA and its environment; the JTPA 
has also determined the basic structure of the statewide service delivery 
system in which SDAs operate. For most states, the JTPA service delivery 
system consists of three major organizations, the state, service delivery 
areas and subcontractors. The numbers of each organization vary con- 
siderably across states. In eleven states no local SDAs exist; the state 
provides all services to participants. [20] SDAs may choose either to 
provide all services or subcontract all or part of this function to other 
organizations in the local area. SDAs providing most services to parti- 
cipants may also elect to establish satellite offices to improve access. 
In some cases, a subcontractor may be a state agency. Due to the co- 
ordination mandate, SDAs (and their subcontractors) are often linked with 
organizations outside the system, most typically local welfare 
offices. [35] Congress has established the roles and responsibilities of 
each organization, v^rhich are summarized in Exhibit 6. [4,5] 

Viewing the SDA as an organization, the state and subcontractors are part 
of its environment. Because most services are delivered to participants 
't the subcontractor level in most states [4, 5, 35], what "goes on" at 
this organizational level essentially determines the services, or 
treatments, that participants receive in the system. [37] Thus, for SDAs 
that subcontract all services, the execution of the SDA's service 
strategy occurs in the SDA's environment and, therefore, to a great 
extent is beyond the SDA's direct control. 
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4/ JTPA outcomes are increased employment and earnings and reduced 
welfare dependency among those who can benefit from and who are most 
in need of JTPA's services. 
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Characteristics of the statewide service delivery system that pose 
challenges to administration and evaluation alike are as follows: 

• Multiple organizations: The state, local service 
del ivery areas and subcontractors form a statewide 
system of employment and training services. 

• Decentral ization: Several organizations (state, SDA 
and subcontractors) are responsible for administering 
JTPA. 

• Local autonomy: Each SDA and subcontractor is free to 
develop its own programs, creating customized forms of 
treatment and implementation. 

• Dynamic programs: Service and implementation 
strategies change over time. 

• Multiple delivery strategies: Many treatments are 
possible to achieve a set of outcomes. 

• Conflicting/multiple goals: Multiple goals may exist, 
some possibly conflicting with each other. 

• Multiple levels of coordination: Administrative and 
service coordination may occur among subcontractors, 
between subcontractors and SDAs, and between SDAs and 
the state. 



In short, the JTPA has established a highly complex service delivery 
system. The more SDAs there are in a state, and the more these SDAs 
subcontract services, the more complex the system becomes. For the SDA 
director with the subcontractor problem, two important insights follow. 
First, because the service delivery system is complex, it will take time 
to sort through the system's complexities to discover the actual causes 
of the problem. Second, the SDA's service strategy, which can determine 
the success or failure of a program, is performed (for the most part) in 
the environment by subcontractors. Among SDA staff, no one really knows 
exactly what these services consist of, nor can they effectively control 
services of the subcontractor having performance problems. 



THE SDA ENVIRONMENT: LOCAL CONDITIONS 

The environment also consists of local economic, social and pol itical 
conditions. How much influence do local conditions have on an SDA's or 
subcontractor's achievement of JTPA outcomes and performance standards? 
Can they account for the subcontractor's performance problems described 
in Chapter 2? There is no information available from the JTPA program to 
answer these questions at this time. However, two separate studies of 
CETA's implementation come to almost identical conclusions. [9, 12] 

• The demographic composition or social characteristics 
of the community did not determine the kind of indi- 
viduals served. Participant characteristics, in turn, 
did not determine placement success. 
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The level of unemployment at the local level was only a 
mild constraint on the program options open to staff 
and on the level of performance. 

This suggests that SDA performance is likely not foreordained by the 
social characteristics of the community, participant characteristics or 
economic conditions. Instead , differences in local managemen t were the 
most important factor explaining program performance under CETA Thit 
is, what goes on inside the organization— thi ngs thaTTan ~be~readi ly 
influenced by the SDA--tends to have the greatest influence on perform- 
ance. As a whole, these findings confirm that SDA and subcontractor 
operations, or processes, can be altered to improve performance using 
tools such as this process model. 



GOVERNANCE/MANAGEMENT 

With this new insight, the attention of the SDA director shifts to 
governance and management— those who have the authority to alter SDA 
operations to boost achievement of JTPA outcomes and the SDA's perform- 
ance standards. Because of the way Congress authorized SDAs in the Act, 
governance can and does vary from one SDA to another. 

A chief aim of the JTPA is to forge a partnership between local 
government and the private sector. In this pursuit, JTPA outlines major 
responsibilities for local elected officials and for PICs. the chief 
mechanism for private sector participation and forging the public-private 
partnership. Local elected officials help determine SDA boundaries and 
select PIC members. As partners, they jointly select a grant recipient— 
the entity which receives funds from the state and is held financially 
accountable— and an administrative entity to run the day-to-day 
operations of the local program. 

According to the National Alliance of Business' (NAB) nat.onwide JTPA 
survey, about 65 percent of the SDAs selected the local government as 
grant recipients, and local government is the administrative entity in 
57 percent of the SDAs. PICs are grant recipients and/or administrative 
entities in less than 13 percent of all SDAs. [3, 20] These figures 
indicate that local government is the grant recipient and administrative 
entity in a majority of SDAs. 

However, even though PICs are in the minority here, the JTPA (Sec. 103) 
reserves a strong policy and oversight role for PICs in partnership with 
local governments. Who dominates the partnership— or who governs the 
SDA— varies considerably across SDAs. In one study about one-third of 
SDAs were government-dominated (mainly the most rural and the most 
heavily urbanized), one-third were PiC-dominated, and about one-third had 
'shared-authority." But in the last category, the PIC usually took a 
policy leadership role, suggesting that PICs play the major governance 
role in most SDAs. In general, PICs prefer to operate as boards of 
directors, avoiding day-to-day functions in favor of setting broad policy 
directions. In most PICs the business majority did, in fact, shape the 
policies and practices of the PICs. Yet, 90 percent of the SDAs report 
substantial PIC involvement in management issues, primarily in the areas 
of selecting the types of training, selecting service providers, over- 
sight, cutting costs and streamlining* management and accounting pro- 
cedures. Other areas of PIC involvement include public relations and 
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ROLES AND RESPONSIBILITIES OF JTPA ORGANIZATIONS 



STATES (GOVERNORS) 

Establish the State Job Training Coordinating Council (SJTCC). 

Designate Service Delivery Areas (SDAs), approve locally developed 
plans, and distribute grant funds to localities based on formulas 
established in the Act. 

Monitor local program performance, prescribe variations in 
performance standards based on special conditions in the state, and 
award incentive bonuses for exceeding goals (or take action, 
including sanctions, when performance fails to meet standards or 
remains poor). 

Establish and administer a new statewide dislocated worker program, 
a discretionary older worker program, a coordination and special 
services program, and a state labor market information system. 



SERVICE DELIVERY AREAS 
Establish Private Industry Councils. 
Develop Job Training Plans. 

Implement Title II-A and Title IIT (where applicable) programs as 
stated in their Job Training Plans in accordance with performance 
standards and coordination criteria established by the state. 

Monitor program performance of subcontractors (if applicable). 



SUBCONTRACTORS 

Develop and submit proposals for training and employment programs 
contai ned i n SDA Job Trai ni ng PI ans or state trai ni ng programs . 

Implement training programs in accordance with performance standards 
established by the state or SDA and specified in legal subcontract 
agreements. 
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provision of labor market information. In the overwhelming majority of 
SDAs, PICs rely heavily on information and support staff of the adminis- 
trative entity, which is often the local government agency. [20, 35] 

In^ summary, this indicates that effective public-private partnerships 
exist in most SDAs. Within the partnership, the PIC more often than not 
plays the governance role, while the local government (or other agency) 
provides administrative, or management, support and assumes re- 
sponsibility for day-to-day operations. Prior to performing a process 
evaluation, the SDA's governance/management should be identified. This 
task is relatively simple yet important because governance/management may 
be a cause of the subcontractor's problems if those responsible for 
governance and management have not operated the SDA's performance control 
system correctly. 



THE SDA PERFORMANCE CONTROL SYSTEM 

The performance control system is the tool that informs governance/ 
management about the level of outputs (proximate outcomes) and impacts 
(final outcomes) produced by the conversion process. The performance 
control system consists of two parts, the plan and control. The plan 
specifies the desired outputs and impacts, or set of performance 
standards, at some future time. The purpose of control is to determine 
whether the standards have been achieved. Achievements are monitored 
through the Management Information System (MIS). Thus, performance 
control systems are concerned with overall results for given periods of 
time—not with specific actions or decisions. SDAs use performance 
control systems to verify that subcontractors (and/or satellite offices) 
meet their respective performance standards—but not to monitor their 
decisions and program activity. [17] 

In virtually all performance control systems (including JTPA), service 
providers generally have considerable freedom to meet their respective 
standards. This autonomy is important because subcontractors, compared 
to the SDA, are directly involved in service delivery and, hence, are in 
a better position to design customized programs meeting the needs of 
their respective employers and participants. In principle, the more 
services are designed to meet local participant and employer needs, the 
more effective the services should be. And the more effective the 
service, the more likely that performance standards will be met. 

Another reason for subcontractor autonomy is the design of the service 
delivery system itself. SDAs and subcontractors are independent 
organizations joined together through contract agreements. SDAs lack the 
authori ty— as wel 1 as the admi ni strati ve resources — to control most 
actions and decision-making in other organizations. 

A final reason for service provider autonomy is the limitations of the 
MIS. In general, the MIS may be used properly to monitor subcontractor 
performance. However, a MIS does not contain sufficient information for 
administrative organizations (that is, an SDA that subcontracts and/or 
has satel 1 ite offices) to make decisions for service providers. Many 
service decisions are based on information not contained in a MIS. 
Knowing that a national manufacturer plans to open a new plant in your 
community may have a greater influence on decision-making than all the 
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information in a MIS. Further, an MIS often reports aggregated, historic 
information which may be of little use for solving immediate or future 
management problems. Last, most administrative organizations lack the 
time to absorb all the information an MIS offers. So, the wise adminis- 
trative organization knows what it cannot know and lets the service 
providers stay independent. 

What happens when performance standards are not met? Governance/ 
management has three basic means of intervention (assuming that poor 
performance is not caused by environmental constraints, such as a 
downturn in the local economy): 



Cancel or not renew the subcontract; 
Replace the manager; or 

Provide technical assistance/increase compliance monitoring. 



The last intervention is used widely by SDAs to handle subcontractor 
performance problems. This is a useful, potentially effective way of 
handling the performance problem. But the form of the technical 
assistance is critical for this intervention to be successful . 
Typically, SDA technical assistance has been a management strategy, such 
as setting clearer goals or increased monitoring. In other words, a 
common SDA reaction to poor performance is to increase administrative 
pressure-- to tighten the screws. Unfortunately, this can do more harm 
than good. Why is this so? [16, 17] 

First, training and service programs are fundamental to achieving good 
performance. When an SDA attempts to improve subcontractor performance 
by either adding to its administrative or management bjrden or 
controlling service delivery, subcontractor effort previously directed at 
service is now used to perform nonservice (compl iance)-related 
activities. The emphasis may switch from serving participants to 
sati sfy i ng performance standards and state- imposed reporti ng requi re- 
ments. In essence, an ends-means reversal occurs , and actual service 
delivery can suffer as a consequence, likely reducing subcontractor 
performance even further. 

Second, providing services effectively to participants is complex work 
requiring competent staff. If subcontractor staff are incompetent, no 
set of plans, rules or orders can make them more competent. Furthermore, 
these measures often prevent competent staff from doing their work 
effectively. 

In sum, technical assistance that merely tightens the screws (through 
various management strategies) will likely not prove to be an effective 
mechanism for improving performance. Instead, technical assistance 
directed at improving services seems more appropriate, mainly because 
services are the "means" of achieving program "ends" (the performance 
standards and goals of the organization). [7] Services may be improved, 
for example, by increasing the capability of staff, such as by gradually 
replacing incompetent staff with more qualified individuals, increasing 
staff skills (through staff continuing education), or hiring a consultant 



to raise staff levels of performance in specific areas where weaknesses 
may exist. Other mechanisms, such as altering services to increase skill 
gains among participants, may also lead to service improvements and 
ultimately increase performance. 

Performance control systems work best in stable and simple environments. 
For SDAs, this means having a fairly stable, predictable local economy 
that IS simple to understand and monitor. These environments allow SDAs 
and their subcontractors to standardize their services and operating 
systems, creating stability in organizations. Managers often agree that 
satisfying performance standards is easier under these conditions 
Complex and dynamic environments, on the other hand, are less supportive 
of performance control systems if the performance control systems are 
inflexible. Developing performance standards for a rapidly changing 
local economy and "hard to serve" participants is at best a difficult 
task. Performance in such environments is likely determined more by 
events happening outside the SDA than the relatively short-term services 
participants receive. 

Performance control systems also have two major, negative 
characteristics. First, they tend to stifle innovation. Innovation (of 
services) involves risk--a risk that performance standards may not be met 
It the innovation fails. Because performance is the bottom line in most 
SDAs, PICs may be less willing to innovate, particularly when current 
services are meeting standards and achieving organizational goals. Thus 
performance control systems encourage conservative program management. 

Second, these systems emphasize economic consequences but downplay social 
ones. Every major decision that a PIC or SDA director makes has social 
as well as economic consequences for most groups. The economic 
consequences of JTPA are its performance standards— percent employed 
average wage rates and so forth. They are easily quantifiable and, 
hence, are easily integrated into the MIS and performance control system. 
Hence, they often have the undivided attention of the PIC, management and 
most everyone else in the system. 

But there are social consequences to decision-making as well. These 
consequences are often hard to measure and, hence, are rarely included in 
performance control systems. Examples include decisions to orient 
services around the job-ready instead of the hard-to-serve, to ignore 
youth below the age of 18 because their outcomes (or competencies) are 
difficult to measure, or to emphasize placements instead of the gain in 
wages from training. Performance control systems can also produce 
undesired administrative behavior such as enrolling people only after 
they have found jobs or transferring clients among programs to avoid a 
negative termination. When an SDA really applies the pressure, sub- 
contractor management may be driven to ignore social consequences of 
their decisions and to meet performance standards— no matter what. Thus, 
performance control systems direct energy ^nd resources toward achieving 
results measured chiefly in economic terms and may, under the worst of 
circumstances, inhibit social responsiveness and responsibility. 

Is it possible to develop a performance control system that accounts for 
social consequences of decision-making under JTPA? Well, yes and no. 
Although we cannot measure the social dimension precisely in most 
instances, crude measures may be possible. Let's use "need" for services 
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as an example. On a scale of 0.00 to 1.00, an unemployed, single mother 
on welfare with children and no skills or work history might be given a 
"1.00"---having the greatest need. Someone who is unemployed and un- 
skilled might be given a ".50". Someone who is unemployed but has a good 
job history might be given a ".25""that is, having a lower need. Then, 
placements could be adjusted, or "weighted," by these need factors, such 
that a welfare mother counts as a full-placement (1.0), an unemployed/ 
unskilled person counts as a half-placement (.50), and the last counts as 
a quarter-placement (.25). This would mean that, to achieve one place- 
ment, a provider could either place one welfare mother or two unemployed/ 
unskilled persons or four skilled persons. This is just one way of 
including social and economic factors in a performance control system, 
and has actually been tried in various forms by the U.S. Employment 
Service (e.g., the balanced placement formula). 

Returning to the subcontractor performance problem, the SDA director has 
gained some new insights about the problem. Specifically, when the 
subcontractor's performance fell , the SDA' s immediate reaction was 
increased pressure for compliance. The subcontractor's performance never 
improved after that. So, maybe the subcontractor's claims that the SDA 
is the cause of its problem have some truth. 



SUMMARY 

This review of the SDA's environment, governance and performance control 
system has produced the following observations: 



The JTPA service delivery system is highly complex. 
Consequently, the cause of SDA problems and the reasons for 
good or poor performance may not be obvious. 

Because of the complexities, performing a thorough process 
evaluation takes time and resources. 

A key determinant of the SDA's success or failure, the 
service strategy, is performed by subcontractors in the 
environment and, hence, is outside the direct control of 
governance/management. 

Unfavorable local conditions do not necessarily cause 
programs to fail . 

Governance/management can hurt subcontractor performance 
through improper use of the SDA performance control system. 



These insights help us understand the environment that SDAs operate in. 
They also reinforce the notion that SDAs are interdependent organizations 
in a larger statewide service delivery system. This means that sub- 
contractors are not islands; their performance is determined at least in 
part by the SDA. 
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This chapter has provided requisite information for performina the 



45 

32 



CHAPTER 3 

HOW TO EXAMINE THE MISSION COMPONENT 



Elements of the Mission Component 

Mission Component as Interface Between the Environment and the 

Organization 

Review of the BETWEEN and miHlH Conditions for the Mission 

Component 

Overview of the Planning Process 

Documenting the SDA's Performance Standards, Revenue and 

Environment 

Documenting the SDA's Mission or Goais 

Documenting Subcontractor Goais 

The BETWEEN Condition: Comparing Goais, Performance 
Standards, Revenue and Environment for Consistency 

Assessing Treatment Decisions 

Assessing Access Decisions 

Assessing Ailocative Decisions 

Discussion of Ailocative Decisions under i*«ytcon§ra£! § 

Examining the Performance Control Sy&l@fti;„ 

Evaluation of Findings: Reviewing the Eviden^o to Determine 
Whether the BETWEEN and WITHIN Conditions ara Satisfied 
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- CHAPTER 3. HOW TO EXAMINE THE MISSION COMPONENT 



Our process evaluation of the SDA (or subcontract organization) begins 
with the mission component. In this chapter, guidelines are presented 
for examining the mission component of your organization When we 
examine an SDA's mission component, a number of elements must be 
considered, such as the following: 

The environment--its economic, political and social 
conditions, as well as state and federal constraints. 

SDA governance. 

The mission, goals and performance standards of the SDA. 
The planning process. 

Revenue, allocative, treatment and access decisions. 

The method of subcontracting and selection of service 
pvcviders. 

The performance control system and the nature of compliance 
and technical assistance efforts. 

These elements are not independent; they are interconnected. In the 
early implementation of JTPA, when SDAs were struggling simply to get 
their programs up and running, several of these elements and inter- 
connections were probably missing in the mission components of many SDAs 
Now that SDAs have program experience and PICs are assuming stronger 
leadership roles, these elements should be well-established in most local 
areas. Especially in administrative SDAs (i.e., SDAs that subcontract 
an services), we would expect the mission component to be fully- 
developed and dominate other components in the conversion process. 

Our examination of the mission component and its elements is a relatively 
important part of the process evaluation. This is because the mission 
component is the interface between the environment and the other 
components of the conversion process. That is, in the mission component 
the SDA must (1) interpret its environment (e.g., local conditions, 
constraints imposed by the state and Congress, etc.), (2) define its 
purpose in this environment, and thftn (3) design its work, coordination 
and social components to accomplish that purpose. 

If gross errors are made ir; the mission component (Tasks 1 and 2 above), 
these errors are often repeated in designing the other components. So, 
if the mission component is off course, the other components will 
probably be too. 

As described in Chapter 1 the following two conditions must be satisfied 
if an organization is to operate well: 

WITHIN: Each component must be involved in the conversion process. 
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BETWEEN: A proper fit must be achieved (1) between the environment 
and the mission component, and (2) between the four 
components of the conversion process. 

In the process evaluation of an SDA or subcontractor, we review each 
component of the conversion process, checking whether both conditions are 
satisfied. If the two conditions are satisfied in all four components, 
the organization is given a clean bill of health. If the conditions are 
not met in one or more components, the components must be re-designed so 
that all the components satisfy the two conditions. 

The process evaluation always begins with the mission component. We will 
first examine whether a proper fit exists between the environment and the 
elements of the mission component (the BETWEEN condition for this 
component). Then, we will perform an overall assessment of whether the 
WITHIN condition has been satisfied. 

We will use the SDA problem described early in Chapter 1 to illustrate 
the guidelines. We will "walk through" the SDA*s mission component, 
touching on the mission component of the subcontractor as well. In our 
exampl e , the SDA performs recrui tment and i ntake , but trai ni ng and 
employment services are subcontracted. This organizational form is 
chosen because it is the most common across local areas. 5/ Material 
from SDA and subcontractor interviews that we conducted in Washington 
State is presented to illustrate how the evaluation is performed. 



THE PLANNING PROCESS 

In most SDAs, the above elements of the mission component are decided 
through the planning process. At first glance, we might expect that the 
PIC would develop goals and make revenue, al locative and treatment 
decisions in a rational, orderly sequence, such as the following: 



Analyze economic, social and political conditions in the 
environment. 

Develop goals based on local conditions. 

Set performance standards consistent with goals (and, hence, 
local conditions) as well as revenue allocated to the SDA 
(according to formulas in the Act). 



5/ Almost 80 percent of the SDAs are using some form of performance 
contracting for their training services. About 53 percent of the 
SDAs are perf ormi ng some , i f not all, of outreach and i ntake 
services themselves, and about 43 percent of the SDAs perform job 
placement services. These figures indicate that only a smal 1 
percent of SDAs either provide al 1 services or subcontract al 1 
services. Rather, most SDAs tend to have both service delivery 
arrangements, with SDAs most likely to control their own entry 
(outreach and intake) and exit (placement) services. [20] 
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Make allocative and treatment decisions to meet SDA goals 
performance standards and local conditions. 



Decide on the mode of service delivery consistent with 
treatment and allocative decisions. 

Docrment the above steps in the SDA Job Training Plan. 

Administer a performance control system +o manage 
implementation of the plan. 

Evidence presented later in this section and our SDA interviews indicate 
that planning rarely occurs in this manner. Instead, planning often 
resembles the following three-step process: 

Step 1: The SDA receives performance standards and revenue 
allocation(s) from the state based on federal formulas. 

Step 2: Local economic^ social and political conditions relevant 
to program implementation are identified. 

Step 3: Based on Steps 1 and 2, goals are established and 
allocative and treatment decisions are made more-or-less 
simultaneously, each consistent with the other and local 
conditions, and all oriented toward achieving performance 
standards within the SDA's budget. These decisions also 
include the mode of service delivery and the performance 
control system. 

In other words, performance standards and limited revenues drive the 
mission component, as shown below. 

SDA Revenue Local Economic, 

and Social and 

Performance Political 

Standards Conditions 



Goals 

Allocative decisions 
Treatment decisions 
Access decisions 
Mode of service delivery 
Performance control system 



SDA JOB TRAINING PLAN 



IMPLEMENTATION OF PLAN 
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Our aim is not to construct a detailed history of how decisions were made 
in this planning process. Instead, our purpose is to define as clearly 
as possible its elements and their interconnections (or, just as 
important, their absence) and how th';'y relate to the environment. One 
way of doing this is to review the 50A's plan. Most SDA plans reviewed 
in this project contain the information required in the Act (Sec. 104). 
A typical table of contents follows: 



• SDA Goals 

• Analysis of Need 

• Recruitment, Eligibility, and Selection 

• Selecting Service Providers 

• Governor's Coordination Criteria 

• Annual Report to the Governor 

• Local Governance Structure 

• Budgets 

• Fiscal Systems 

• Monitoring Systems 

• Signature Page 

• Appendices 

A quick review of this list indicates that the plan includes many of the 
elements of the mission component. So, why not just review the plan to 
determine the relations between the elements of the mission component? 
There are several reasons why this should not be done, such as the 
fol lowing: 



SDAs often develop plans primarily to obtain funds; what 
goes on in the SDA may not resemble what's stated in the 
plan; 

Planning is a continuous process, not a one-shot activity. 
Plans developed in one year may not apply in a later year; 

The first plans of many SDAs often lacked a mission 
statement or goal s , reflecting the infancy of JTPA. This 
information must be obtained from other SDA sources; and. 

Although sections in SDA plans may be well developed, their 
interconnections are rarely stated in clear terms. 
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In short, a variety of sources—the SDA plan, other documents as well as 
staff— are needed to determine the makeup of each element and their 
interconnections within the mission component. Guidelines for doing this 
are described in the next section. 



PERFORMANCE STANDARDS, REVENUE AND ENVIRONMENT 

Let us begin with information easily obtained in most SDAs: performance 
standards and revenue. For purposes of discussion and illustration, we 
assume that the SDA has performance standards equal to the national 
standards set by DOL in the early months of JTPA's implementation. In 
addition, the state and PIC have set standards for specific target 
groups. Like most SDAs, ours only has sufficient revenue to serve 
five percent of its eligible population. [34, 35] The SDA's performance 
standards and revenue allocations should be documented and, for now, set 
aside for later use in this chapter. 

The next task is to identify local conditions— economic, social and 
political. The local labor market is an important element of the 
environment. Most SDAs have already documented local labor market 
conditions in their job training plans. If not, many states can provide 
such information through labor market information systems developed under 
JTPA. Common indicators of local market conditions and trends are listed 
below. 



INDICATORS OF LOCAL MARKET CONDITIONS 

LABOR: 

Unemployment rate 

UI claims experience 

Percent of population on welfare 

Jobs with increasing demand 

Jobs with declining or stable demand 

MARKET TRENDS (Growth vs. Decline): 

Number of building permits issued 
Number of new businesses since 12 months ago 
Number of business closings and bankruptcies 
since 12 months ago 

A second environmental factor affecting the SDA mission is social 
conditions. In general, these refer to the social characteristics of the 
local area, the predominant social problems in the area, and the 
distribution of these problems across groups. Common indicators of 
social conditions are listed below. 



INDICATORS OF SOCIAL CONDITIONS 

SOCIAL CHARACTERISTICS: 
Population, total 
Population for specific: 
age groups 
race/ethnic groups 
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sex 

income per capita, household— overall 
or by key target groups 
education levels 
rel igious groups 

employment statuses by key target 
groups 

Di stri buti on of the above characteri sti cs i n 
different areas of the SDA. 

SOCIAL PROBLEMS: 
Crime rates 

Incidence of alcoholism/drug abuse 
School dropout rates 

Percent of population below poverty level 
Percent of population on welfare 
Number and percent of households with a female 
head 

Incidence of serious health problems 

In many cases unemployment is just one of several problems that a 
particular group may face. Knowing how the problems are distributed in 
the area and, in particular, among the eligible JTPA population and 
applicants is essential for designing services that can properly meet 
individual needs. 

The third environmental factor is political conditions. Normally, 
indicators of these conditions are qualitative, such as the following: 



INDICATORS OF POLITICAL CONDITIONS 

Local government - business relations (i.e., can a 

public-private partnership work?) 
The legacy of CETA in the local area - local 

government and business role in and support 

of the CETA program 
Number of jurisdictions SDA serves (i.e., must 

services be distributed fai rly across 

jurisdictions?) 
Influence of local government(s) on SDA operations 
Infl uence of local business on SDA operations 
Political influence of subcontractors 



These 1 ists are incomplete, but are representative of dimensions that 
prof i 1 e the area. CI early , some i ndi cators wi 1 1 be more important i n 
some SDAs than others. Whatever the indicators actually used, the aim 
here is to construct a prof i le of the relevant economic, social , and 
political conditions facing the SDA. For now, set this profile aside for 
later use in this chapter and move on to the discussion of SDA goals. 
For thi s di scuss ion , assume a scenari o wherei n we f i nd that our SDA 
primarily serves a large city with an unemployment rate above the 
national average. Applicants often have multiple barriers to employment. 
The large city has a gradual ly decl ining economy. Several contracting 
agencies exist to provide services to participants. Some of tiiese 
agencies are more favored than others by local government. 



UNCOVERING THE SDA'S MISSION 

The next task is to identify the SDA's goals and to understand how thev 

aoats'rr'- t'^"'^^^ an SDA either L easily identT^i^M s'oJ n^ 
goals at ail. In some cases the goals are included in the SDA's inb 

Tt^:%?l7ut%T'' '''' statements'VsUblls el 

oy tne pic but, for one reason or another, not included in the olan 

the^l^ thTs^A-s ^e-co^ds.^^'^ ^^^^ Jher^L^J^-d 

Injxamining an SDA's goals, the following factors should be kept in 

An SDA may have a list of goals which it attempts to achieve 
over a given period. A number of mo-e detailed objectives 
may be associated with each goal. In this goal-objective 
structure, a goal is achieved when most of its objectives 
are satisfied; 

Alternatively, an SDA may have a general "mission statement" 
that states the overall thrust, or direction, of the agency 
Normally, the mission statement is accompanied by one or 
more goals, each perhaps having its own, more detailed 
objectives; and.. 

The common mission of all SDAs is to match participants with 
employers in the labor market. Most SDAs mention this 
common mission in their goals and mission statement. 
However, the emphasis that an SDA places on employers versus 
participants often varies. Some SDAs orient their programs 
around employers. Some orient them around participants 
Other SDAs try to give equal emphasis to employers and 
participants. Congress has clearly stated that three 
outcomes of the JTPA connection are increased employment and 
earnings and reduced welfare dependency among participants 
most in need of services. 

The relative emphasis given to participants and employers is often quite 

fSnhl! -'Vf^^' and mission stateirients. Here is a mission statement 
emphasizing the participant: 

A. To provide comprehensive employment and training services required 
to prepare and place eligible SDA residents into unsubsidized 
employment. Specific emphasis will be placed on selecting employ- 
ment and training opportunities which will increase the earned 
income of program participants and will result in secure, full-time 
unsubsidized jobs. 

Contrast, that statement with the following employer - (labor market-) 
oriented mission: 

B. To assist local busine.SEes solve employment-related business 
problems, allowing both business and individuals to increase 
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productivity and prof itabi 1 ity. To support local economic 
devel opment efforts by the preparati on of 1 ow- i ncome , unempl oyed 
area residents as a workforce for new or expanding business and 
industry. 

The following mission statements indicate equal emphasis on both 
employers and participants: 

C. To provide permanent unsubsidized job placements for JTPA 
participants through cost-efficient and effective employment and 
training activities; and to provide business with a motivated and 
able workforce in occupations for which there is actual demand or in 
occupations for which business perceives near term shortages. 

D. To provide services that match the employment needs of the 
participant with the needs of the local labor market. 

E. To provide private sector and unsubsidized public sector placements 
among JTPA participants through effective and cost efficient 
activities mutually beneficial to both employer and participant. 

Acknowledging the powerful influence of performance standards, some PICs 
have developed goals centered around performance standards: 

F. To operate the SDA in the most effective and efficient manner 
possible. 

G. To develop and implement a system of program activities to meet or 
exceed all established performance objectives. 

Once you have identified your SDA's goals, check to see whether they are 
consistent with each other. For example, a goal to operate the SDA in 
the most effective and efficient manner possible may be incompatible with 
Goal A (above), which can imply a lower level of efficiency. Such goal 
inconsistencies may create potentially serious problems in the work and 
coordination components as service delivery is steered in conflicting 
di recti ons. 

These goals are often called the organization's "manifest" goals — that 
is, they are the formal, well-documented goals of the SDA. But an SDA 
may have other informal, or latent, goals as well. You cannot find these 
in the SDA's formal documents (such as its job training plan); they must 
be inferred from how resources are used and how management and staff 
allocate their time. For example, an SDA may have a manifest goal of 
providing comprehensive services but instead mainly provide low-cost 
services that meet employer needs (its latent goal). In general, an 
SDA's latent goals can only be discovered through an exan-,ination of the 
other elements of the mission component and the remaining three 
components oF the conversion process (in Chapters 5-7). 

In some cases, this exercise may reveal that the SDA has no clear, 
well-defined goals. Unless the SDA is operating in a highly unstable 
environment, this indicates an organization adrift, having no clear 
direction or purpose. This result should be documented for later 
reference in the process analysis. 



To illustrate this discussion, our archtypical SDA has two goals, A and 
emphasizing participant-oriented, multi-component services and the 
JTPA connection. Before moving on, however, it is also necessary to 
document the goals of the subcontractor. This is particularly important 
for an SDA experiencing subcontractor problems, such as ours 



DOCUMENTING SUBCONTRACTOR GOALS 

SDAs typically contract with a wide variety of organizations—public 
schools, private industry, the employment service, private training 
organizations, colleges, community-based organizations and others For 
some, providing JTPA services is tlieir single purpose. For others, JTPA 
service delivery is just a small part of what they usually do. If the 
subcontractor is c local college or university for example, its goals 
are likely qui + j different fro-^ those of the SDA. In most cases, the 
goals of the college are oriented around the edur ional needs of the 
community rather than the specific training needs of JTPA participants. 
Because of such goal disparities, colleges might not provide training 
customized for the JTPA participant. In fact, rather than having 
contrccts with local colleges, SDAs usually refe: participants to 
appropriate classes where openings exist. [35] In short, the less 
dependent an organization is on the SDA for its existence, the more its 
goals may differ from the SDA's goals. And as its goals differ, so will 
its programs. However, if SDAs and providers are enrolling the most 
job-ready applicants into their programs, such differences may not be 
critical. 

In contrast, the more an organization is dependent on JTPA funds, the 
more its goals may reflect those of the SDA. In our example, the sub- 
contractor falls into this category, and we find its goals to be fairly 
similar to the SDA's goals. This indicates that both organizations 
should be operating in consistent directions. 



THE BETWEEN CONDITION: A COMPARIfON OF GOALS, PERFORMANCE STANDARDS 
REVENUE WITH THE ENVIRONMENT ' 

Now it is time to retrieve all the information we have collected and 
determine whether the SDA satisfies the BETWEEN condition of the mission 
component. Our focus is to check whether the SDA's goals, performance 
standards and revenues are consistent with each other, as illustrated by 
the "consistency triangle" in Exhibit 7. Its three key questions are as 
fol lows: 

QUESTION 1: Are the goals, performance standards and revenue 
consistent with the environment (arrows 4-7 in 
Exhibit 7)? 

QUESTION 2: Are the goals and performance standards consistent 
with each other (arrow 1)? 

QUESTION 3: Are revenues adequate to accomplish the SDA's goals 
and performance standards (arrow? 2 and 3)? 



EXHIBIT 7 



THE "PERFORMANCE STANDARDS - GOALS - REVENUE" 
CONSISTENCY TRIANGLE 




EXHIBIT 8 



MISSION COMPONENT: 

COMPARISON OF GOALS, PERFORMANCE STANDARDS, REVENUE AND LOCAL CONDITIONS 



LOCAL CONDITIONS 



ECONOMIC 



SOCIAL 



POLITICAL 



GOALS 



PERFORMANCE 
STANDARDS 



SDA 
REVENUE 



High un- 
employment 
Decli fling 
^ ecorioiny 
•^Declining 
local , 
td), base 



Urban population 
Welfare rates 
Poverty levels 
Average education 
Racial mix 
Drop-out rates 
Assessment of 
applicants' 
training and 
other needs, 
which are often 
multiple 



Working public- 
private partner- 
ship exists. 

Active PIC 
sets SDA 
policies 

Business 
majority 
controls the 

Local Govt, 
support for 
subcontractors 



Goal A 

- Comprehensive 
services 

- Long-term 
unsubsidized 
employment 

- Increased 
earned income 

Goal D 

- matching needs 
of participants 
with employer 
needs 



DOL national 
standards 



Allocated according to 
Title II-A formulas. 

Sufficient to serve 
about five percent of 
eligible population. 

Small amount of revenue 
from incentive funds 
and local contributions. 



Good SDA- 

business 

relations 
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In answering these questions, you may find it helpful to arrange your 
information in columns as shown in Exhibit 8. Because all the relevant 
information is side-by-side, comparing one column to the next for 
consistency is easier. To address the three questions above, we must 
begin outside the triangle in Exhibit 7 with local conditions and federal 
and state requirements in the environment. 

QUESTION 1 . Ideally, local conditions and the SDA*s goals should be 
consistent with each other, as the following examples illustrate: 

• ECONOMIC: If a strong economic development program (through 
the local Chamber of Commerce) exists in a local area, the 
PIC might establish an SDA mission oriented around economic 
development that emphasizes employer needs. Alternatively, 
an SDA with low unemployment rates might anticipate problems 
in recruiting participants, so it may develop a mission 
oriented around marketing quality participant and employer 
services with employment guaranteed. Finally, SDAs in 
dynamic, changing local economies may not have any specific 
goals. In such an unstable environment, labor market trends 
might be monitored closely, with programs changing as 
changes in the market occur. 

• SOCIAL: A PIC in a rural SDA might adopt quite different 
goals than a PIC serving a largely low income, minority 
population in a central city. For example, in one rural 
SDA, 27 percent of the eligible population consisted of 
migrant, Hispanic farm workers. For this group, training 
represented a major change in lifestyle. The worker must 
settle in the area long enough to receive training, then 
stay more-or-less permanently in the community as a member 
of the local labor force to achieve long-term job retention. 
To address tho needs of this group, the SDA established 
goals oriented toward the participant, offering longer-term 
training and emphasizing long-term retention of employment. 

• POLITICAL: Given the following business' views in one SDA, 
it is not surprising that the PIC adopted goals supporting 
labor market needs: 

"We must give business people the power to make human 
resource decisions without undue influence and navigate them 
through the paperwork to get them good employees in a way 
which is cost effective for their business." 

(The SDA should aid local economic development efforts by 
providing training money to companies wanting to relocate 
here and training and tax incentives for young or expanding 
businesses to assist them with cash flow needs. ) 



In short, choosing an SDA*s goals is always a choice of values. And 
these choices usually involve judgments about local conditions and how 
the JTPA connection does or should work in a given community. For 
example, contrast the situation in which employers make job offers, so 
services should be oriented. primarily toward their neer^s versus a 
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iI^H?i r 1 services to reduce participant needs, thereby 

Irill ?K^'"Pl°^-'"'"^' .Usually, the PIC. who has the power and authorit^ 

to make the choices, decides what values should be part of the SDA's 

mission and goals. ^ ^ 

In comparing the local conditions with the goals of our SDA (Exhibit 7) 
we find they are generally consistent with each other. The high un- 
employment rates and multiple social problems in the eligible population 
indicate that goals oriented around comprehensive services and the JTPA 
connection are justifiable. 

' wll' check the other side of Exhibit 7. the consistency between 
federal/state requirements and the SDA's performance standards and 
revenue (arrows 6 and 7). These will usually be consistent because they 
are determined by revenue formulas in the Act (Sec. 202) and the DOL 
regression formula for SDA performance standards (that virtually all SDAs 
use). Thus federal authorities have largely determined two parts of the 
IV^^K^'cn?^ standards. DOL has established what the performance 

?nA^^K c ono^'^.^^- appropriating funds to JTPA (and allocated to 
bUMs by Sec. 202). Congress has decided how much money SDAs shall have to 
meet the standards. If the standards are high and the appropriations are 
low (arrow 2). SDAs are forced to provide participants with low cost 
services that are capable of meeting the standards. 

The consequences of this federal influence on the SDA's mission component 
depend on the SDA's goals. For example, an SDA might have a very low 
unemployment rate and goals oriented around econonic development. In 
this case, the SDA's goals and local conditions are consistent with the 
federal revenues and performance standards; all are geared for high- 
volume, short-term training and placement services. A "proper fit" has 
been achieved among the elements of Exhibit 7. 

QUESTIONS 2 AND 3. In contrast, our example SDA has high unemployment 
and poor economic and social conditions; its goals are oriented around 
comprehensive services. Here the federal oerformance standards and 
revenues are inconsistent with its goals and local conditions. The SDA 
must provide short-term services to clients needing long-term training. 
To achieve a "proper fit" between goals, performance standards and 
revenues, the SDA must either lower its standards or increase its 
revenues (or both). 6/ Unfortunately, the SDA controls neither of these 
two environmental factors, so it cannot make these adjustments. 7/ So, 
in terms of the two questions raised earlier, we find that in our SDA: 



6/ This would increase the average amount of funds per participant, 
making long-term programs possible. In addition, notice that 
changing goals to favor labor market needs may not be a viable 
option, for then goals and local conditions would be inconsistent. 

7/ However, many SDAs have the regression formulas for calculating 
performance standards, and they are gradually learning how to 
manipulate the standards through program adjustments. 
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• Goals and performance standards are inconsistent (Question 2). 

• Revenues are inadequate to' accomplish the SDA's goals (Question 3). 

When the federal-local forces in Exhibit 7 are in conflict, the common 
result is that goals are ignored as performance standards dominate the 
mission component. Set these findings aside for now, and let's move on 
to collect other information about the mission component. Later, we will 
use this information to determine whether the WITHIN condition is 
satisfied. 



TREATMENT DECISIONS 8/ 

Against this back-drop of revenue, local conditions, goals and 
performance standards, the treatment decisions in the SDA should be 
documented. This is often a relatively easy task. The SDA plan is a 
good source for identifying allowed services or treatments. The plan 
should be cross-checked with current practices to determine whether 
discrepancies exist (e.g., the plan states that 15 percent of parti- 
cipants will receive OJT, but in reality less than five percent receive 
the service). You should also identify all services that participants 
receive through coordination agreements with outside agencies. Hov/ever, 
our main focus is on the following issues: 

Consistency : Are the SDA's local conditions, goals and performance 
standards linked directly with treatment decisions? That is, are 
the four consistent with each other? 

Dominance : Do performance standards primarily dictate treatment 
decisions? 

Service Strength : Is the service appropriate for the SDA's population 
and powerful enough to be expected to have some effect if it is 
delivered as outlined in the plan? 

Connection : How does the JTPA connection work? 

Consistency refers to whether a "proper fit" exists among the above 
elements, while dominance reflects the realities of JTPA: regardless of 
the needs of an SDA's participants and employers, performance standards 
and low revenues often push the SDA toward designing short-term (low- 
cost) treatments in the mission componen^t. The extent of consistency and 
dominance in an SDA may be readily determined by comparing the training 
needs of enrollees with the number of enrollees in multi-service 
programs. Performance standards usually dominate treatment decisions in 
SDAs that uniformly provide one-shot, short-term services to most partic- 
ipants, irrespective of their needs and employment barriers. On the 



8/ Vou may recall that through its treatment decisions, the PIC is 
determining the service strategy of the agency. 
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other hand, the presence of a system for assigning one or more services 
to participants based on some criteria (such as in competency-based 
programs) indicates that performance standards may influence but likely 
do not dominate treatment decisions in the mission component. 

Next the underlying assumptions, or service strengths , behind these 
treatments should be documented. That is, among the 28 services allowed 
under the JTPA, why are some services provided and not others? Does each 
service promise long-term job retention? How much service is needed for 
most participants to gain long-term employment? 

Finally, the services that match participants with employers in the labor 
market should he documented. What percent of participants receive these 
services? What percent find jobs? What percent of employers who request 
referrals actually hire a participant? These yield information about how 
the connection is working. Answers to these questions are collected 
through the MIS and through discussions with SDA planning staff and PIC 
members most responsible for making treatment decisions. 

In short, you should not only describe the services offered but also how 
these services— when delivered in the prescribed manner— lead partTc^ 
ipants from unemployment into employment (or other positive outcome). In 
our SDA, a variety of services is offered to meet the variety of needs 
among participants. Services are provided in a competency-based job 
training system (described in greater detail in the next chapter). All 
training programs (which may be either single or multiple component) are 
designed to achieve measurable and certifiable competencies that are 
required in the local labor market. A substantial portion of staff time 
IS devoted to job development. A majority of participants find jobs 
through job search assistance programs. 

This indicates that our SDA has mechanisms for matching participants and 
employers. However, treatment decisions suffer from dominance (i.e., 
performance standards and revenue dictate that most participants receive 
low- cost services, regardless of need). As a consequence, while a 
majority of participants need (higher-cost) training, only a minority 
receive this service. This reduces the service strength in our SDA. 
Thus, while the SDA's placement services that a majority of participants 
receive result in high placement rates, they may not be of sufficient 
strength to result in long-term employment (a goal of our SDA), as might 
be expected under a training program. 



ACCESS DECISIONS 

Once treatment decisions are made, the PIC must decide who will receive 
the SDA'.< services, but only in part. In designing JTPA, Congress has 
established eligibility and target group requirements to restrict access 
to scarce resources. States have also established service requirements 
for numerous other groups (e.g., ex-offenders, displaced homemakers, 
migrant farm workers, etc.). The eligibility requirements define who can 
and cannot enter the system; the target groups define individuals of 
special interest in this eligible population. Because SDAs normally have 
performance standards associated with their target groups, they have 
incentives to recruit individuals with these charactersitics. 
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In principle, the purpose of the eligibility and target group 
requirements is to act as screening/sorting mechanisms to assure that 
services are provided to those who are most in need. In most cases, 
however, they apparently do not fulfill this purpose. [34, 35] Within 
any given eligibility or target group, there is often a range of "job 
readiness," or need for employment and training services. Faced with the 
pressures to meet performance standards within a limited budget, SDAs (or 
subcontractors, whoever are performing outreach and intake) have in- 
centives to select eligibles who are the most job ready (i.e., those who 
are easiest — and, therefore, less costly — to place). 

To offset this tendency to "cream," our SDA has added "unemployment" to 
its list of eligibility requirements, hopefully to restrict access to the 
most needy applicants. This access decision is consistent with the SDA's 
goals and local conditions. In the next chapter we will examine whether 
this additional requirement actually works as the PIC intended. 



ALLOCATIVE DECISIONS 

Next, the PIC must also make al locative decisions as part of the planning 
process. That is, the PIC must decide "who gets what" in the SDA, such 
as: 

• How much of each service to offer (i.e., the allocation of 
training funds among the services authorized in the Act). 

• What services to provide in-house, through satellite 
offices, or to subcontract. 

• What coordination arrangements (e.g., with welfare agencies) 
to enter. 

• Which agencies/subcontractors to fund. 

By their nature, these are political decisions. [9] They involve the 
weighing of competing claims on SDA revenues by local interest groups 
(such as subcontractors), each seeking decisions in its favor. Here, 
political constraints in the environment become paramount. SDAs which 
fail to satisfy the demands of competing interest groups may find their 
operati ons perf ormi ng def ens i ve hoi di ng maneuvers ( i n response to 
interest group pressures) rather than performing integrated service 
delivery related to local needs. 

As described earlier, allocative decisions are rarely a separate, 
isolated task in the SDA' s planning process. In practice, treatment- 
allocative decisions are usually made jointly as staff and PIC members 
try to strike a balance among such factors as: 

• Service needs of employers and participants. 

• SDA performance objectives. 

• Amount of funds tc be al 1 ocated and the cost of each 
service. 




The performance records and political influence of 
subcontractors. 

The different services that different contractors provide. 
* Local conditions. 

Simultaneous with these jaint treatment-all ocative decisions. SDAs must 
alsa make allocative decisions regarding what services to provide in- 

IaH^tv°%'"^""*'^''\c^V9"""''"^' subcontract services for a 

yanety of reasons. (See Appendix B far a brief discussion of this 
issue.) O^ce an SDA- has. decided to subcontract all or part of ils 
services, subcontract allocative and treatment decisions (that is. what 
subcontractors provide how much of what services) are made jointly' based 



• The number of local subcontractors. 

• The services they, are qualified to provide. 

• The participant groups they usually serve. 
*" Their track records. 

Local political pressure (to fund a specific sub- 
contractor(s). regardless of past performance). 

• Other factors. 

These subcontract al locative-treatment decisions are often made on two 

I 6\/6 I Si 

Method of Subcontracting : SDAs must decide how training 
funds and services are allocated among subcontractors. Two 
basic choices exist, market and function . A market 
allocation means the subcontractor serves some group of 
clients. Examples of market allocation include sub- 
contracting by target group (i.e.. youth, adults, handi- 
capped, etc.. are each, served; by a separate subcontractor) 
or region (e.g.. in a multicounty SDA. each county has its 
own subcontractor(s). who serves all eligibles in its 
jurisdiction). Combinations are possible, such as a sub- 
contractor providing all youth services in a county (i.e.. 
market allocation by group and region). ' 

Allocation by function means dividing funds among sub- 
contractors by service and. occupation. For example, an SDA 
may issue 88 contracts among 17 subcontractors to provide 
training in over 50 different occupations. The type of 
training varies in each contract. 

SDAs sometimes mix market and function allocations, such as 
issuing a contract to provide youth participants with JSA. 
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Selection of Service Providers : Once the PIC decides on the 
method of subcontracting, it must decide which local service 
providers receive what contracts. And it must also decide 
what services--if any-«the SDA is to provide, and how these 
services are to be coodioated with subcontractors. 

Ideally, both of these decisions— the method of subcontracting and the 
selection of service providers— are driven by SDA goals. For example, an 
urban SDA orienting its programs around the diverse needs of local 
employers might choose a functional allocation of services (to achieve 
services specialized to meet the needs of specific employer and partic- 
ipant groups), which in turn influences later service provider 
selections. 

In reality, however, PICs carefully chose their service providers as the 
primary mechanism for meeting performance standards. L35] Most PICs 
chose subcontractors based on past performance (placements and cost) 
under CETA. ^ Organizations that either served "high risk" individuals 
(thDse^ requiring multiple support services and seriously lacking work 
histories and skills) or operated multi-component programs were not 
favored. Given the new PIC criteria for service provider selection, most 
agencies simply changed their focus to short-term, low-cost and high- 
placement operations that did not focus on particular groups. As the 
head of one agency commented, in order to get a JTPA contract it had 
changed its orientation from "...taking the tough cases, to becoming an 
efficient personnel office for local business." [35:23] In short, the 
PIC mission (of satisfying performance standards) also became the mission 
of the providers. Because most SDA-provider contracts had payment 
contingent on performance (mainly through the use of performance-based 
contracts), accountability was passed from the SDA to the provider for 
services rendered. 

But the transfer is not total accountability, only partial 
accountability. Many SDAs issue contracts through an RFP process. 
Particularly under function subcontracting, an RFP may specify the number 
of participants to enroll, what services they should receive, and how 
many must be placed in jobs (or have other positive outcomes). By sub- 
contracting in this way, the SDA retains control over these allocative 
and treatment decisions of the mission component. The provider's main 
task is to implement these prior SDA decisions formalized in the 
contract. In this way, even though an SDA may subcontract all of its 
services, all of the SDA's accountability is £iot transferred to the 
provider. In general, SDAs are accountable for the group-level 
allocative, treatment and access decisions embedded in the contract, 
while providers are responsible for meeting contract performance 
standards through its access, treatment and allocative decisions made on 
a person-to-person basis in its day-to-day operations. This division is 
more likely under functional than market subcontracting. In the latter 
case, subcontractors may be given greater freedom to decide how its 
training funds are allocated among the services authorized by the 
contract. 

Finally, this transfer of accountability may differ by the subcontract 
method. In function subcontracting, other elements of the SDA mission 
component, such as the organization's goals and planning process, are 
relevant to the SDA--but net to the provider--and, hence, are usually not 



51 65 



' suSrnnJ;^r;n Comprehensive process evaluation of all the SDA's 

subcontractors, similar analyses must be repeated with ei^F nnp Thd 

5e1crib?n 'th^Ti-IIioS '^^^"^ 't' constructing' o^f or".\V sulary taM 
2!!^ni? ^ miss on components of the subcontractors. Exhibit 9 is an 

yorlp crfrc''^ed';'\'° ""'^''^ ^'"^^'"^^^^ the table to uU 

your specific needs. Or, you may wish to construct a separate table fnr 

each subcontractor, listing its features in greater detan fs'uch a^^ tZ 

r f??eTf'an SOrVl- 1" V'''' ''''' ta'bleTpVovi e'a u ef' 

proTiie of an SDA s subcontractors. Comparisons among subcontractors ran 

subcontractor meets ,ts performance standards, the subcontractor oerfor- 

s? Sda d's" Thrs'?/""- J^^n'?''"' '"^ sSA^ perf rm'a ce 

stanaards. This is an indication of faulty allocative derisinnc in +h« 

m,ss,on component, which can be corrected thr^gh "bette; manVemeSJ' 

If an SDA chooses to subcontract all or part of its sprviVoc +um n« 
functions are added to the mission component Fi-rst the SDA n^Vst des a^ 

prl^'a' sTrom'"i'''/'°"" announcing service 'coHtracJs' ece 
proposa s from local service providers in response to announcements 
evaluating the proposals and awarding contracts. To perfom ?his 

wh?c "supp^on^^th^'Vlc' i^ separate •'coJtTlcTunU:- 

wMiLH support the PIC in selecting service providers Second as 

TX^^:^^^^' ^-^-^ ^' -"o^^'ratl 

tiiI"sl'°s"'iVlc" s->^an7. '•'^'•^ "^^d ^° ^'^'^ whether 

tne bUA s (PIC s) allocative decisions are consistent with the SDA's 

goals, performance standards, and revenue (from Exhibit 7) and its 

treatment and access decisions. The following are examples of all ocat We 

decisions consistent with the other elements of the mission compone ^ 



An SDA in a low unemployment area adopts goals emphasizing 
the needs of the labor market. High performance standards 
and low revenues push the SDA toward short-term treatments 
with low costs. A majority of the SDA's revenues are 
allocated to these services; and. 

An SDA in an area with high youth unemployment also has high 
youth performance standards and relatively high youth 
revenues under the 40 percent rule. Therefore, revenues are 
evenly divioed among training programs and placement 
services. 
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EXHIBIT 9 
SDA SUBCONTRACTOR SUMMARY 



PERCENT 

CLIENTS AVG. 

*JHO AVG. AVG. STAFF 

TOTAL NUMBER RECEIVE COST COST TIME 

TYPE OF TOTAL JTPA JTPA NUMBER EACH PER PER WITH 

NAME CONTRACTOR REVENUES REVENUES STAFF CLIENTS SERVICE CLIENT SERVICE CLIENT 
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In our example SDA, relatively low revenues and higii performance 
standards have resulted in allocations supporting lower cost treatments, 
such as job club and other placement services. Thus, the dominance of 
performance standards and revenue over goals in our example SDA (as 
discussed in Exhibit 7) has directly influenced treatment and allocative 
decisions. Treatment and allocative decisions are consistent with 
performance standards and revenues but are inconsistent with the SDA's 
goals. 

Performance standards also dominate the mission component of our youth 
subcontractor experiencing performance problems. Through the subcontract 
mechanism, ^ the PIC has essentially transferred the pressures associated 
with meeting performance standards down to the subcontractor level. 
Thus, although the SDA and subcontractor have similar goals, performance 
standards dominate the mission components of both organizations, pushing 
them in parallel directions. 



PERFORMANCE CONTROL SYSTEMS 

Not long ago, the Boy Scouts of America revealed that 
membership figures coming in from the field had been 
falsified. In response to the pressures of a national 
membership drive, people within the organization had vastly 
overstated the number of new Boy Scouts. To their chagrin, 
the leaders found something that other managers have also 
discovered: organizational control systems often produce 
unintended consequences. The drive to increase membership 
had motivated people to increase the number of new members 
reported, but it had not motivated them to increase the 
number of Boy Scouts actually enrolled. 

Cammann, C. and D. A. Nadler. "Fit Control Systems to Your 
Managerial Style." Harvard Business Review. January 
1976:65. 



SDAs that contract services or have satellite offices usually operate 
performance control systems to monitor sub-unit activity and performance. 
These systems must be assessed because of their potential to push the SDA 
in the wrong direction, as illustrated in the Boy Scout example. For 
administrative SDAs, monitoring, compliance, technical assistance and 
related activities become the center of attention, for these are the 
SDA's "means" of achieving its "ends," or its performance standards. As 
a consequence, compliance and related activities often intensify, whether 
they are truly needed or not, which, for the reasons cited in Chapter 2, 
often lowers subcontractor performance. 

Our purpose is to assess the scope and intensity (what elements are 
monvtored and at what frequency) of the SDA's performance control system. 
To make this determination, the monitoring procedures of the SDA's 
contract unit shoiHd be collected. Subcontracting agencies should also 
be interviewed to determine monthly hours they devote to monitoring/ 
compliance activities and whether, in their view, these are excessive. 
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Systems which extend beyond the limited number of required areas— such as 
performance standards and verifying expenditures, accuracy of records and 
coordination agreements—are probably overextended, are consuming 
excessive administrative resources in the SDA, are likely haviicg a 
negative consequence on subcontractor performance, and should be cut back 
in scope. In our example SDA we find that excessive compliance may be 
one cause of the subcontractor's recent poor performance. 

EVALUATION OF FINDINGS 

In this section, findings from our review of the mission component are 
evaluated to determine whether the WITHIN and BETWEEN conditions are 
satisfied. The major findings are summarized below: 

• The archetypical SDA in our discussion serves a large urban 
population with an unemployment rate above the national 
average. The area has a gradually declining economy. 
Applicants have multiple employment barriers. 

• The SDA has two goals: provision of comprehensive services 
leading to secure, unsubsidized jobs; and matching the 
employment needs of participants with the needs of the local 
1 abor market. The goal s are consi stent with local 
conditions. The subcontractor experiencing problems also 
has similar goals. 

• The goals and performance standards of the SDA are 
inconsistent. Revenues are inadequate to accomplish the 
SDA' s goals. 

• As a consequence, performance standards dominate the SDA's 
goals and push the organization toward short-term, low-cost 
services, even though most participants need long-term 
training. In essence, achieving the performance standards 
is a latent goal of the SDA. 

• SDA revenues are allocated to support sl^ort-term, low-cost 
services. Thus, the SDA's performance standards, revenues, 
and treatment and allocation decisions are consistent with 
each other. This same pattern also holds at the sub- 
contractor level . 

• The PIC has added "unemployment" as an eligibility criterion 
to reduce the demand for services and to assure those most 
in need are actually served. Access decisions are, there- 
fore, consistent with goals. 

• Excessive SDA compl iance efforts may be one cause of the 
subcontractor's poor performance. 

This evidence suggests the following conclusions regarding the WITHIN and 
BETWEEN conditions: 
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WITHIN CONDITION: The condition is satisfied in part. The SDA (and 
subcontractors ) performance standards , revenue , al 1 ocati ve 
decisions and treatment decisions are consistent with each 
other, but, as a whole, are inconsistent with the SDA's (and 
subcontractor's) goals. 

BETWEEN CONDITION: The condition is satisfied in part. The SDA's 
goals and access decisions are consistent with local 
conditions, but its performance standards, revenue, al locative 
decisions and treatment decisions are not. However, the latter 
are consistent with federal and state requirements. 

In short, the SDA's (and the subcontractor's) performance standards, 
revenue , al 1 ocati ve deci si ons and treatment deci si ons domi nate ^he 
mission component. Instead of meeting the needs of the local population, 
the SDA's services are designed to accomplish performance standards. As 
a. direct consequence, while the SDA will likely achieve its standards, it 
may probably fail to achieve its goals. 

This pattern in the mission component may be widespread in JTPA. In the 
first nine months of JTPA, the pattern of local program priorities, 
services and participants was more one of similarity than diversity. [35] 
In essence, the program constraints discussed previously— inadequate 
funding, performance standards, spending limits and others— were driving 
SDAs to make similar program choices. Two performance standards, place- 
ments and cost,^ were viewed as critical among PICs, and most programs 
were designed with an eye toward achieving these ^"1 the other standards. 
Thus, although 28 training activities are authorized under the law, most 
SDAs focused heavily on two activities, classroom training and on-the-job 
training. In contrast, the special needs of eligible individuals, 
support servi ces , i nnovati ve and exempl ary programs , mul ti-servi ce 
programs, and customized services for specific target groups all received 
considerably less attention and emphasis in designing programs. In 
effect, the constraints have standardized service delivery to favor labor 
market needs vs. the hard-to-serve. 

Officials at 80 percent of the SDAs said they made no 
concerted effort to define or serve those "most in need" and 
"able to benefit from" JTPA"one of the act's targeting 
requirements— and did not believe that either the legis- 
lation's provisions or its level of funding were conducive 
to dealing with those eligibles who most needed assistance 
in securing jobs. The lack of commitment and effort to this 
aspect of the act resulted in, according to three-fourths of 
the SDAs , substanti al screeni ng by servi ce del i verers to 
enroll those eligibles who were most job-ready. The 
extensive screening, plus the modest level of support and 
remediation services, and the unexpectedly brief duration of 
JTPA training--!! weeks for adults and !2.2 for youth- 
coupled with the high placement rates, did cause concern 
that many participants would have gotten jobs without the 
benefit of JTPA. [35] 

This implies that Congressional design of the JTPA's implementation 
strategy may be having unintended consequences. Specifically, although 
Congress created performance standards to assure local accountability 
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(Appendix A), perhaps the chief effect of performance standards is to 
push the system to serving the most job-ready eligibles in service 
delivery areas. 

For now, put these findings aside, and let us move on to the work 
component. 
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CHAPTER 4 
HOW TO EXAM^E THE WORK COMPONENT 
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Definition of the WITHIN and BETWEEN Conditions for the Work 

Component 
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CHAPTER 4. HOW TO EXAMINE THE WORK COMPONENT 



I!;at''s!;oulTrM?J SDA's service strategy-the "treatments" 

se^iices cLncpH J" long-term, unsubsidized employment. How are these 
services supposed to work? Answers to this question begin with a basic 
understanding of education and employment in'the United States which s 

IxmiTna th. P''^^^"^^ guidelines f^r 

examining the work component in SDA and subcontractor organizations. 

EDUCATION AND EMPLOYMENT IN THE UNITED STATES 

^nMnf"*,""^'""!! systems have the primary responsibl ity for preoaring the 
?nS?^^d.^■''^' /'"^^ "'-^^■"^■"9 ""'--"^l^y occurs beyond an 

DHmaH^v th K"'''^?°:"^' '"^ ^ advancement occurs 

primarily through job mobility and cumulative work experience. For those 

Tn\h° It through the educational system and onto a job ladder 

nff.nl options become quite limited. These individuals 

often become the labor market's casualties: 

...Those with limited skills, experience, and credentials 
the victims of stunted opportunities, discrimination, and 
bad luck, the residents of poverty areas and declining labor 
markets, and those whose individual, family, or cultural 
proMems^^ undermine successful performance in the work 

l""^ P'^^^ram mechanisms for dealing with the labor 
market s casualties has been a major federal issue for the past 25 years 
Historically, Congress has responded to this manpower problem through a 

np«I?nnn,o /''°H • ^^7^ Redevelopment Act (1959), the Manpower 

Development and Training Act (1962), the Economic Opportunity Act (1964) 
the New Careers Program (1965). the Work Incentive Program (2967) the 
Job Opportunities in the Business Sector Program (1968), the Concentrated 
Employment Program (1968), the Emergency Employment Act (1971), the 
Comprehensive Employment and Training Act (1973), and--Congress ' current 
Ennnir"^ Training Pa -Grst>ip Act (1982). By and large, the 

fundamental, underlying assumpcions behind most of these programs have 
not changed greatly over the years [31], which are as follows: 

• The skills and attitudes for this segment of the labor force 
are malleable and subject to intervention. 

• Education and training is a desirable and effective 
intervention for increasing the productivity of these 
individuals in economic markets. 

• The problems of this group will not be solved by an improved 
economy alone. 

• Once provided education and training for earning a living, 
an individual may become a productive member of the labor 
force for his/her lifetime. 
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Given these assumptions, Congress has authorized training, education, 
remediation, ^ support services, and non-occupation activities for youth 
and adults in JTPA. These services produce intended results through a 
"chain of objectives." [23, 30] All programs are created to accomplish 
certain purposes as defined by the program's objectives. The objectives 
of virtually all programs are temporally interdependent; some objectives 
must be achieved early, others later, forming an orderly sequence, or 
chain, of activity. Objectives in the chain are ordered such that each 
objective builds from the one before and becomes a necessary condition to 
the one that follows. The distinction between objectives and means of 
achievement becomes relative; each objective in the chain is a means to 
achieving the next objective in the chain. 

Exhibit 10 presents JTPA's chain of objectives and validity assumptions. 
The general objectives for participants appear on the left-hand side of 
the table, while the underlying assumptions linking the objectives appear 
in the middle. These assumptions define beliefs about cause-effect 
relationships in the program as participants progress from entry to exit. 
The right column indicates some of the key employer entry points in the 
participant chain— that is, places where participants are commonly 
matched with employers. 

Perhaps the most startling point of Exhibit 10 is how little control SDA 
staff have over the chain and the client/employer connection. Individual 
and employer participation is voluntary. Both are free to drop out of 
the program at any time. Employers ultimately decide who is hired, which 
may occur at any point along the chain—prior to entry, during treatment, 
or at exit— though the last is probably the most common. Staff really 
control only two major parts of the chain: entry into the SDA and the 
services participants and employers receive. 9/ These are captured by 
the access and treatment decisions, respectively, in SDA process model. 

In the next section, guidelines are presented for examining the work 
component in an SDA or subcontractor. 10/ The guidelines are based on 
the hierarchy of objectives in Exhibit 10; their purpose is to determine 
whether the work component satisfies the WITHIN and BETWEEN conditions. 
For the work component, the WITHIN condition requires that (1) the work 
component is involved in the conversion process, and (2) that the parts 
of the work component are consistent and interrelated, working in 
synchrony to achieve positive proxin.ate outcomes. 



9/ Staff control over services is probably not total control. For 
example, under CETA clients who could find OJTs got them, and 
clients requesting specific classroom training often received it. 

10/ For SDAs that subcontract al 1 services, the work component is 
performed mainly by the subcontractors. 
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EXHIBIT 10 

A HIERARCHY OF PROGRAM OBJECTIVES AND VALIDITY ASSUMPTIONS IN JTPA 



HIERARCHY CF PARTICIPANT OBJECTIVES 
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I. Ultimate Objectives 

1. Accomplish JTPA purposes as 
specified in the legislation. 



Achieve positive finajioutcomes 
(employment retention, increased 
earnings, welfare dependency, 
performance standards, etc.). 

Achieve positive proximate outcomes 
(placement, increased earnings, 
welfare dependency, performance 
standard, etc.). 



II. Intermediate Objectives 

4. Receive credential 
documenting achievements. 



5. Complete training (treatment) 
program. 



Learn new skills, change 
attitudes, improve employability 
skills consistent with aims of 
program and participant needs. 



7. Determine appropriate program 
(treatment) for participant as 
allowed under JTPA, 



8. Ditermlne participant needs. 



VALIDITY ASSUMPTION LINKAGES 



SOME POINTS WHERE CLIENT/EMPLOYER MATCHING OCCURS 



Long-term employment of participants 
resolves nation's manpower problems 
as defined by Congress. 



JTPA services enable participants to 
retain employment over the long-run. 



Completion of program and credentials 
will enable participant to find employment. 



Credentials are needed to enable participant 
to compete In labor market. 



Programs cause behavioral and attitude 
changes only when completed. 



Skills, attitudes and behavior are 
malleable and can be chanted by JTPA 
services. 



Treatments have greatest i?iipacts when 
targeted to remove an individuars 
employment barriers. 



An IndlviduaVs employment barriers 
can be determined reliably. 



Employer hires participant. 



Employer hires on-the-job training participant 
contingent on successful completion of program! 
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VALIDITY ASSUMPTION LINKAGES 



SOME POINTS- WHERE CLIENT/EMPLOYER HATCHING OCCURS 



Services must be targeted to those who 
qualify to achieve ultimate objectives 
and performance standards. 



Employ. T refers individual (who may or may m^t be a 
employee) to JTPA for training. Individual i: 
enrolled itr eligible. 



Program must have adequate supply of 
appropriate individuals to achieve 
ultimate objectives and performance 
standards. 



Employer needs a specific number of individuals 
trained to perforni a specific task(s). A 
training program is developed to meet the 
employer's needs, or qualified participants are 
referred through placement/job search services. 
SDA may recruit individuals or draw from applicant 
pool to meet employer needs. SOA may recruit 
employers to participate in OJT, -institutional 
training or other programs. 




The BETWEEN condition requires that the work component be consistent with 
the mission component. The remainder of this chapter mainly consists of 
guidelines examining the WITHIN condition. In a concluding section we 

SnSJLc ?1J "":P°1!"t satisfies the WITHIN and BETWEEN 

conditions If both are satisfied, we may conclude that the work 
component is operating properly. 

WITHIN CONDITION 

In Exhibit 10 the hierarchy of objectives describes a general sequence of 
cw ^ pathway, as participants progress from entry to exit through 
the system. The specific characteristics of pathways differ from one SDA 

Z^hT. ^ ^? 1"°^^^'- ^^^^^ contract all services have no 

pathway; it can be found at the subcontractor level. To illustraf^ our 

rJ^Mhir?^ ^^^^r/ °^ ^"^ presented on the next page 

ci?nh? P' borrowed from a model actually used in one SDA but modified 
slightly for this chapter, 11/ Tc follow these guidelines in your own 
organization you must first describe its pathway. 12/ The issues raised 
in this chapter should apply to most SDAs and subcontractors, not lust 
the one presented here. ^ 

The guidelines begin with recruitment (Objective 11 in Exhibit 10) and 
move through the pathway to the client's status at follow-up (Objective 2 
in Exhibit 10). Examination of the WITHIN condition focuses on the 
to 11 owing two factors: 

Whether the steps in the pathway are consistent with each 
other. Steps should be structured to provide clients a 
logical sequence of services from entry to exit. 

The flow of clients through the pathway. Not everyone who 
enters the pathway necessarily completes it. Some drop out 
at various points; others make it through all the steps. By 
analyzing the flow of clients from one step to another, we 
can make inferences about where problems lie in the system. 
Then, other, more specific information can be collected to 
determine v;hat the problem is and how it should be 
corrected. 



11/ The pathway is for an SDA performing outreach and intake but 
subcontracting training and employment services. 

12/ This is done by first identifying all of the steps a participant may 
possibly go through from entry to exit, and then connectir.ij thp 
steps to show all possible routes that a participant may traverse 
through the system. 
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EXHIBIT 11: PARTICIPANT PATHWAY 
(with hypothetical client flow numbers) 



Walk- ins 

Telephone Inquiries 
Etc. 



1. Recruitment (est. : 500) 



Non-JTPA 
Services 
(45) 



Educational 



2. Application Submitted (220) 
3. Eligibility/Verification Interview (220) 



;^7fji2flibje_ 



Eligible 
4. Assessment (175) 



\ 

5. Individual Training Plan (ITP) 

and Support Payments (122) 
(Enrollment) 

I • 

6. Subcontractor Job Training Services 

-1 — ^ — 



i\6ferrals 

and 
Coordinatic 
Links 



1 



Work Maturity 



Pre- Employment 



Occupational Skills 



Remedial Education 
Basic Skills 

Training 
GEO 
ESL 

Support Service 

I 



Workshops 
Job Counseling 
Work Experience 
Support Service 



Workshops 
Job Club 
Job Counseling 
Job Search 
Assistance 



7. Employment or Other Positive Outcome (98) 

I 

8. Follow-up Services 



Skill Training 
OJT 

Try-out 

Work Experience 
Customized Training 
Customized Training 

I 



9. Follow-up Status (64) 
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In our discussion we use the flow of youth through the pathway in the SDA 
and subcontractor organizations. Mduiwdy m ine bUA 

RECRUITMENT (STEP 1) 

III ^^"^^^ ^i^P- ^° examine recruitment services in the pathwav. 
M??^'?! "ly.'^i'^^e 3n adequate supply of clients to achieve their 
ultimate objectives and performance standards. In most SDAs the demand 
n^L'J''^'u^' exceeds resources, so few outreach services are usually 
needed. Where outreach is performed, it is often targeted at specific 
nard-to-find participant groups, such as high school dropouts or older 
workers, that have low (walk-in, or "passive") application rates. SDAs 
and c:r.tractors have developed different strategies for performing 
outreach, depending on tr.e group, such as the following: Penorming 

• Establish h>^anch offices to improve access to JTPA services; 

Move intake to where clients frequent (e.g.. settino-up 
information and application tables in high schools' to 
recruit y?'jth); 

Establish coo :i nation/referral linkages with other local 
agencies, such as AFDC; and. 



Encourage employers to refer employees for traini 



ng. 



Yet, depending on environmental constraints, even the best outreach 
er torts can fail. For example, in one state few female ex-offenders 
entered JTPA because the female state prison operated its own employment 
and training program. More commonly, the application process may contain 
major barriers that dis-.tiade people from ever applying, such as the 
following: 

• A social security card is required. 

Applications are only in English. Staff speak only English. 

Application require- military draft status for male youth. 

Parents of youth are reluctant to provide family income and 
other information required to determine eligibility. 

In terms of the pathway in Exhibit 11. the number of clients applying for 
services (Step 2) from all sources (recruitment, referral, etc ) in a 
period should be counted (using the MIS). 13/ This provides baseline 
intake volumes. Then, these baseline volumes can be broken into specific 



13/ The MIS must be designed to store all eligible and ineligible 
applications in a period to perform this calculation. The "period" 
may be defined as the previous quarter to minimize data storage 
requirements. 



target groups and compared with performance standards. If the number of 
applicants is lower than the performance standard for the target group, 
the standard can never be achieved. In essence, more applicants satisfy- 
ing the standard must be recruited. This involves expanding current 
outreach efforts (at a minimum), eliminating barriers in the application 
process, or changing services altogether to increase their attractiveness 
to potential clients. For illustrative purposes, the numbers in 
parentheses in Figure 1 describe the flow of 500 youth applicants through 
the pathway in the previous quarter. About 220 of these youth apply for 
services. At this rate of entry, all of the SDA's youth performance 
standards for the year would be satisfied. 



ELIGIBILITY (STEPS 2 AND 3) 

The next step in both the pathway (Exhibit 11) and the hierarchy of 
objectives (Exhibit 10) is eligibility. At rtV^s point, our aim is to 
determine how the SDA*s eligibility requiremj^ntt* reduce the number of 
applicants in the pathway. Eligibility standards are defined in the 
JTPA, although states and SDAs may alter their severity by changing the 
definitions of some terms (e.g., the family). Cnr example PIC has added 
an^ eligibility requirement to the Act*s, unemployment, as a way of 
objectively rationing scarce resources in areas of high demand. For the 
most part, once eligibility standards are established, determining 
eligibility becomes an objective process that should not vary from one 
intake staff to another. If procedures in your SDA reveal that whether a 
client is eligible depends on who interviews him or her, the procedures 
should be changed so that all staff apply the same (objective) criteria 
and, therefore, reach the same eligibility determinations. SDAs may 
program their MIS or other computer to determine applicant eligibility in 
a standard, accurate manner. 

Periodically, a random sample of applications should be cross-checked to 
verify that the client's eligibility was correct and, therefore, that the 
intake process is reliable. Many SDAs also regularly include cross- 
checks to verify the accuracy of information that applicants provide. 
For example, one subcontractor specializing in ex-offender services finds 
that sometimes applicants with no prison record say they are ex-offenders 
simply to receive free services. 

In our example SDA, 175 applicants, or about 80 percent of those applying 
for services, were eligible to receive them in the previous quarter. The 
20 percent of applicants who were not eligible were referred to non-JTPA 
services. 



ENROLLMENT, ASSESSMENT AND ASSIGfs; cNT (STEPS 4 AND 5) 

The next step is to examine the SDA's intake process, which consists of 
the SDA's "pigeonholing process' and the access and treatment decisions 
for specific applicants. Each these is reviewed below. 

ACCESS DECISIONS . These decisions control the flow of individuals and 
employers, primarily, and other resources into the SDA. For individuals, 
access to OTPA services is a two-step process. First, the individual 
appl ies (either passively or through recruitment) to the program, and 
eligibility is determined. Second, if the applicant is eligible, he or 



she may or may not be enrolled in the program, depending on the followi 
factors. 



Creaming to meet performance standards . Applicants who are 
eligible have different education, training, histories, 
attitudes and motivations. Some are more qualified than 
others for employment. Faced with the pressures of meeting 
performance standards, a majority of SDAs enroll eligibles 
who are most job-ready. [20, 35] 

* Recruitment to meet performance standard s. SDAs often 
allocate their resources based on progress toward satisfying 
performance standards. If some standards have been met, 
resources are often shifted to other (unmet) standards. If 
an applicant does not contribute to the unmet standards, he 
or she may be ignored (i.e., put on a waiting list) and 
eventually drop out. 

* Limited funds . On average, SDAs have funds only for about 
five percent of their eligible populations. [4] If appli- 
cant demand for services exceeds resources, some eligible 
applicants may not be served or may not receive the training 
they really need. [19] Conversely, if the supply of re- 
sources exceeds the demand for services (e.g., such as when 
too few youth apply for the 40 percent youth funds), SDAs 
wi 1 1 more 1 i kely accept doubtful , i f not all, c'^ i ents . 

* Sorting/ screening . Between application and enrollment, many 
SDAs require eligibles to participate in orientatio.i and 
assessment sessions. These sessions help sort out the less 
motivated applicants, who often drop out prior to enroll- 
ment. 

Front-line staff Jjave substantial influence over participant access 
decisions. For the eligible applicants, they act as "gatekeepers," 
deciding on a case-by-case basis who does and does not receive se:^vices 
These enrollment decisions are often higfily discretionary and are in- 
fluenced by PIC policies, planning decisions, tl,e above foCtorr , as well 
as participant preferences. 

The important point here, however, is that access decisions are 
controlled (but not totally) by the SDA and, hence, can bj manipulated to 
improve achievement of the SDA's performance standards and goals as well 
as JTPA outcomes (i.e., increased employment and earnings and reduced 
welfare dependency for those most in need). 14/ However, in a previous 



14/ Some SDAs subcontract their access decisions. Faced with pressures 
to meet performance standards , subcontractors often intensively 
screen--or cream--el igibles to enroll the most job-ready persons, 
thereby reducing the risk of possibly failing to meet their perfor- 
mance standards. As Walker and his colleagues report [35:27], "In 
one SDA, a proprietary school reportedly turned down 25 JTPA 
eligibles for every one accepted; in another, a bank teller training 
program screened 118 JTPA eligibles to get 19 enrol lees." While 
such screening may assure that most enrol lees are placed, it also 
may lead to shortfalls in overall enrollment. 
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CETA study placement success was not related to participant 
characteristics. [9] The hard-to-serve could be given the highest 
priority and programs still performed well. Similar trends exist in 
JTPA, such as the ability of most SDAs to meet welfare performance 
standards. [35] 

In our example SDA access decisions are consistent with its goal to serve 
eligible residents. (See Chapter 4.) The official policy of the SDA is 
tc serve everyone who is eligible, but also to use the intake process to 
"sort out" the motivated applicants from the less motivated ones. The 
SDA relies heavily on the design of its intake services to achieve this 
effect. For example, the SDA requires that virtually all =»pplicants 
attend orientation, assessment and assignment sessions, plus complete 
"homework," such as obtaining reference letters and practice filling out 
employment forms. If an applicant does not help the SDA meet an un- 
satisfied performance standard, he or she is placed on a waiting list 
before going through assessment. Not all applicants have sufficient 
motivation to complete the intake process. For others a waiting list is 
an inconvenient, if not a major barrier. These individuals select 
themselves out of the program prior to enrollment. Thus, the intake 
process is used to reduce the number of applicants eligible to receive 
training and placement services, to achieve a highly motivated group of 
enrollees (whom employers prefer), and to develop an individual training 
plan customized to each person's needs. 

Yet, faced with pressures to meet performance standards, the SDA 
sometimes bends the rules. For example, when an employer refers an 
individual for training and a placement seems virtually guaranteed, the 
SDA may skip the formalities of the intake process and move the person 
directly into training. This also happens when the placement rate for 
specific target groups is too low. Applicants who satisfy a target's 
standard may bypass the hurdles of the intake process, have an ITP 
developed, and swiftly enter into training and placement services. 

PIGEONHOLING PROCESS . Either before or after an applicant is enrolled 
(depending on the pathway in a given SDA), services are assigned to the 
participant through a "pigeonholing process ." [16] This process varies 
across SDAs but usually consists of the following elements: 

• Services offered to participants in a SDA are selected (ir 
the planning process) from the list authorized by Congrei^s 
(Sec. 204, 205; see mission component). 

• The services are transformed into a set of standardized 
programs. Each program is a set of skills that staff stand 
ready to offer participants and employers. 

• When a participant enters the office of the SDA, his or her 
needs are defined from such personal characteristics as 
previous education, work history, etc. 

• The participant's needs are categorized, or placed in a 
pigeonhole—that is, also standardized. 
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The participant's category of need, in turn, dtcermines what 
program the participant receives (diagnosis). In most SDAs, 
only one (training) service is assigned usually per parti- 
cipant, mainly because most PICs view mul : i-service programs 
as too costly. And if PICs are mainly enrolling the most 
job-ready, multi-service programs may br, unnecessary anyway. 

• The participant receives the program. 

In other words, by standardizing services and participant needs, 
pigeonholing simplifies service delivery tremendously. Staff no longer 
have^ to develop customized services to meet the unique needs of each 
participant. Diagnosis--or matching participant needs with programs that 
el irpi nate them-- i s al so standardi zed ; parti cipants i n the vari ous 
categories of need (or pigeonholes) are assigned programs in a pre- 
determined way. All of this saves resources in the service delivery 
process, 

FcT the pigeonholing process to wcrk well, the following conditions must 
satisfied: 



yjfngnosis must be standardized . Loose but nonetheless 
"predetermined guidelines must exist for matching participant 
needs with programs. Th^ase guidelines should be applied 
consistently across staff ar*d applicants. 

* Staf f re quire ski lis to p^ f -rfcrm diagnosis . Assigning 
p.irticTpants to program^ based on need is a discretionary 
process Judgment is required in defining a participant's 
needs and pigeonholing him or her into a given category. 
Participants often help out by categorizing themselves. 

• The environment must be complex and stable . Compl exi ty 
arises throicgh the diverse needs of participants. Stability 
i s requi red for staff skills to become wel 1 -def i ned , or 
standardized, into formal programs. 

^ T he process pi.us t be consistent with SPA (foals . 

Two key aGsumplions of this pigeonholing process are that an individual's 
employment barriers can be assessed, and that services have greatest 
impacts when tavrjeted to remove an individual's employment barriers. 
Therc^ is no single, c.^r-ect method of assessing and assigning services to 
a person. Rather, a ariety of methods exist, all requiring considerable 
discretion and staff expertise to be implemented properly. 

T REATMENT DECISIONS . In this pigeonholing process treatment decisions 
-'4re vvc<er the control of staff and, hence, can be manipulated to improve 
perl'o. .jiance. Two elements seem to be key for achieving high performance, 
first, stc^^f must ^ave adequate skills to perform diagnosis and operate 
programr. Indeed, the quality of staff was perhaps the most important 
de"^;erminant of performance under CETA. [9] Second, and central to the 
assumptions in Exhibit 10, the programs assigned to participants through 
t! ^ pigeonholing process must effectively eliminate or reduce 
pn- ticifjants' needs. They are the intervention that's supposed to make 
the JTPA connection work. If participants are misdiagnosed (that is, 
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their needs are categorized incorrectly), or if the wrong programs are 
assigned for a given category of needs, or it^the program itself operates 
poorly, poor performance may result. Most Pigeonholing processes do not 
run properly all the time; the above problems are common in most service 
delivery systems. At best, pigeonholing is imperfect, and it is up to 
management to handle the disturbances and conflicts that normally arise. 

In our example SDA, a competency-based pigeonholing process is used. The 
process is well-documented, and a rev!es' of participant records reveals 
that it is implemented faithfully. The chief purpose of intake is to 
document client achievements and deficiencies in the following four 
competency areas: 

Educational : Possessing the besic computational and communication 
skill levels necessary for successful entry into the labor market 
for thoir occupational choice; 

Wo i'- k Maturity : The demonstreted ability to apply positive working 
ha-'its which meet employer expectations and fulfill basic employee 
responsibilities SuCh as reliability, motivation, ability to carry 
out instruction, etc.; 

Pre-employment : A basic av/areness of the world of work, knowledge 
of work opportunities, skills for selecting among these 
opportunities, and the job search skills necessary to secure and 
present well in informational and job interviews; and, 

Occupatiouc^T SkilU : Mastery of entry level and advanced skills 
required for a specific occupation or class of occupations defined 
by employers, uro:uns, or generally accepted occupational standards. 

Paper and pencil tests, staff evaluations, completion of assignments and 
other diagnostic tools are L.sed to benchmark competencies at entry. Once 
established, the necessary sequence of services to overcome deficiencies 
i::. assigned (through the ITP) as shown in Step 5 of Exhibit 11. 

Before closing our discussion of the intake process, its overall design 
must be considered. Up until this point, the order of events in the 
pathway (Exhibit 11) and the hierarchy of objectives (Exhibit 10) have 
been the sme. In this stage they differ, as shown below (where 
"assignment/- means assigning services to the participant through the 
individual training plan): 



Enrol Imert occurs first in the hierarchy of objectives but last in the 
pa hway. In the hierarchy of objectives, the SOA enrolls everyone who is 
eligible; in the pathway the SDA enrolls only eligible adults and youth 
who successfully complete assessment and assignment. By placing enroll- 
ment last in the sequence, the SDA increases its influence over access 
decisions, or who receives training and placement services. The infor- 
mation reviewed in this volume and our discussions with SDA and con- 
tractor staff indicate! -ihat most SDAs enroll clients according to the 
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Assessment 
Assignment 
Enrol Iment 
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process shown in the pathway (Exhibit 11). By placing enrollment last in 
the flow, SDAs are in a better position to cream. Furthermore, it also 
can give the SDA greater control over its performance. If training and 
placement services are subcontracted, SDAs doing intake can alter their 
performance levels simply by changing referrals to providers that, for 
whatever reasons, place better. 

summary, even though intake is not part of the job training services 
clients receive, it plays a critical role in determining the success of 
the program. Our SDA's job training services may be highly effective in 
eliminating deficiencies in the four competency areas. Yet, if a 
client's competencies and deficiencies are not properly identified, or if 
deficiencies are matched improperly with services, the effectiveness of 
these services may ba reduced greatly. Thus, the twin purposes in this 
part of the pathway examination are: 

• To determine whether your SDA has a systematic, accurate way 
(or pigeonholing process) of determining client needs and 
assigning services based on these needs. 

To udtermine what percentage and what kinds of clients go 
through what steps in the intake process. 

• To determine how the intake process affects the flow of 
clients from application to enrollment. 

Returning to Exhibit 11, in our example SDA about 30 percent of the 
eligible applicants drop out during intake, leaving 122 enrol lees (with 
15 percent of these bypassing normal intake procedures). Thus, of the 
220 persons who applied for services, about 55 percent received training 
or employment services. Further breakdowns of enrol lees into target 
groups indicate that performance standards for some groups may not be 
met. Therefore, the SDA has two basic options to increase the number of 
enrol lees: either increase recru.tment of applicants or relax the 
screening devices of the intake process. Rather than modifying a 
well-working cog of the organization, the SDA elects to bolster its 
recruitment efforts. 



TREATMENT (STEP 6) 

In our SDA, once the participant's ITP is completed, he or she is 
referred to a subcontractor, where job training services are provided 
(Objectives 4-6 in Exhibit 10 and Step 6 in Exhibit 11). Supposedly, 
these are the critical events that will make the difference, that will 
move the participant into employment (or other positive outcome). In 
general, the effectiveness of these services depends on two factors. 
First, the servi;^ must actually reduce or eliminate the participant's 
needs and/or c af iciencies. In general, all JTPA services require the 
participant tn change in some way. Institutional training, for example, 
is designed to increase a participant's skills, which in turn sho Od 
increase his or her chances of finding unsubsidized employment. If the 
training fails to increase a participant's skills, his or her chances of 
finding employment may not be improved by the service. Thus, the 'ink 
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between a JTPA service and a job is indirect. Whether or not a service 
results in unsubsidized employment depends on the service's success in 
changing the participant as intended during treatment. 15/ 

Therefore, in order for the subcontractor or SDA to judge good services 
from bad ones, the provider must be able to measure both the kind and 
amount of change in a participant that a service (or sequence of 
services) has produced. 16/ For example, a youth entering a word 
processing course may already have typing skills but know nothing about 
computers. After completing the course, the participant is a skilled 
word processor. In the participant's institutional training course the 
gain in skills was measured by (1) the participant's demonstrated ability 
to perform specific word processing tasks; and (2) an overall grade for 
the course. This gain in skills led to a job shortly after the course 
was completed. By recording such skill gains in the MIS, the SDA and 
subcontractor can readily detect services that are effective or in- 
effective in changing participants in the manner desired. In this way 
good services can be perfected while bad services can be improved or 
eliminated. As a whole, changes in participants (such as a gain in 
skills) produced by JTPA services are an intermediate objective of the 
program (i.e., Objective 5 in Exhibit 10). 

The second factor influencing service effectiveness is how the service is 
implemented. Here the primary issue is service integrity , or whether the 
services that are formally prescribed to participants are actually given 
or experienced by them. The issue typically occurs when (1) the 
participant receives fewer services than prescribed in his or her ITP, or 
(2) the subcontractor substitutes one service, say classroom 
training (CT), for another, say placement services. The first situation 
can be determined by periodically examining client case folders for time 
spent in each service (or an MIS analysis could be conducted if the dates 
of service are in the data base and the MIS service data are fairly 
accurate). In the second situation, the contractor has a performance- 
based contract, and its interest is to place as many participants as 
cheaply as possible. Therefore, it strives to place as many participants 
through low-cost placement services rather than its higher-cost CT 
program. At times, participants may be officially recorded as receiving 
CT in the MIS, but in reality they enter employment through placement 
services. The participant's folder may show that he or she received some 



15/ This reasoning is implicit in the JTPA. Paraphrasing the Act's 
statement of purpose (Sec. 2), JTPA is to provide training to youth 
and unskilled adults to increase their skills so they may, in turn, 
obtain productive employment. 

16 / Please remember that these services may be provided by either JTPA 
organizations or outside coordinating agencies. 
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training, but this amount is usually far below the SDA average, does not 
increase the participant's occupational skills, and is not responsible 
for the participant's employment. In essence, the program is a 
masquerade where placement services (emphasizing counseling) are made to 
look^ like training. While these services undoubtedly improve the 
participant's ability to find a job (because most everyone can get a job 
with their current skills), they often do not help the participant hold a 
job because no training to improve skills was provided. In these cases, 
we might expect a subcontractor to have high placement but low retention 
rates. 

All of this indicates that subcontractors can manipulate the MIS to their 
own benefit and, hence, the SDA cannot rely fully on its performance 
control system to find out what services participants are actually 
receiving. Sporadic audits of contractor case files may be necessary. 
But the TS can be appropriately used to compare placement and retention 
rates; m.j'or discrepancies here might suggest the substitution of 
placement services for training. With the increasing emphasis on 
retention in JTPA, one way SDAs may counter this masquerade (or 
" service creaming " ) is to make reimbursements based primari ly on 
retention in their contracts, which is consistent with the ultimate 
objectives of JTPA (Objectives 1 and 2 in Exhibit 10). Another measure 
is to set a ceiling on the number of participants placed through direct 
placement in the contract. In either case, most SDA staff may know 
little of what actually goes on at the service provider level. 

There are many ways of providing services to participants, such 
competency systems [2], wjrk adjustment hierarchy systems [22] and 
others. The relative merits of these alternative approaches are unclear, 
mainly because we know very little about how services should be designed 
to achieve the best outcomes [7]. For now, however, we will use the 
above two factors for determining the effectiveness of any given service 
strategy. 

0:ir SDA's competency-based service strategy seems to satisfy these two 
criteria. Ongoing assessment is used to measure the participant's 
progress in el iminating deficiencies- The participant's Individual 
Training Plan (ITP) specifies when his or her progress will be monitored 
during treatment. The plan also identifies the agency responsible for 
providing training services and the competency gains to be achieved. 
Gains in competency are measured through a set of standard indicators 
developed for the four competency areas and their respective subareas. 
For example, some of the categories (and their respective indicators) for 
work maturity are as follows: 

Rel iabi 1 ity: 

is on the job regularly with minimal absence 
is prompt to work and returning from breaks 
reports to work in good condition 
notifies supervisor when absent or late 
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Motivation: 



does not avoid any part of the job 

seeks additional work when assigned tasks are completed 

asks questions and seeks opportunities to learn 

shows enthusiasm and volunteers extra effort when 

necessary 

In short, whether the service actually reduces or eliminates needs is 
determined through the regular, on-going monitoring of the participant's 
progress through his or her ITP. If needs are not being addressed by the 
prescribed services, the plan can be altered to increase service 
effectiveness. Where such progress evaluations are a normal part of the 
training program, process analysis of the SDA's treatments becomes a 
day-to-day ^operation. 

Normally, such monitoring systems are developed to follow an individual ' s 
progress. But sometimes an entire training program offered by a sub- 
contractor may be bad. Usually, these programs or agencies may be 
readily detected by their relatively high dropout or low placement rates. 
Using its performance control system, the SDA or contractor closely 
monitors pathway dropout and placemen": rates to detect problems. But the 
MIS is of little use in pinpointing the actual causes of flow problems. 
Where high treatment dropout or low placement rates exist, the contractor 
needs to evaluate the quality of its program and determine whether 
feedback mechanisms exist to assure participant needs are addressed--that 
the first factor above is satisfied. In these cases, program quality may 
be upgraded by hiring more qualified instructors, increasing the length 
of training, improving the training provided to participants, or some 
other measure. 

TREATMENT UNDER SUBCONTRACTING . Before leaving the treatment stage of 
the pathway, which many SDAs subcontract, the consequences of sub- 
contracting on treatment effectiveness should be considered. The chief 
benefit of subcontracting is the potential for specialization, which may 
lead to hipher levels of performance than when SDAs deliver services 
alone. Thai is, when an SDA divides service delivery by market or 
function, subcontractors can de-^ign services customized for specific 
participants in specific training areas. This usually has a positive 
influence on the effectiveness of services, boosting performance levels. 
Of course, subcontracting also has undesirable consequences on service 
delivery, and these are mentioned here. 

First, depending on the method, subcontracting may adversely influence 
access decisions. When services are provided either by the SDA or by 
other organizations, the client still decides whether to participate in 
the program. However, when an SDA subcontracts al 1 services by function 
(i.e., subcontracts are for providers to train participants for specific 
occupations), the client, through his or her choice of the subcontractor, 
also chooses the services that he or she receives. This is because, 
under JTPA's performance-based system, few incentives exist for one 
subcontractor to refer appliciints to other subcontractors in an SDA. 
Instead, when a job-ready and eligible client walks through the door of a 
subcontractor, we can be fairly sure that client will be enrolled in the 
subcontractor* s program, even though other subcontractors in the SDA 
might be offering services more suited to the client's needs and 



interests. This is less likely if services are subcontracted by market 
so that, for example, one provider is responsible for alj youth, or all 
adults, or alj handicapped participants. Here, there is only one source 
of services for each participant group. If a youth walks through the 
front door of the adult service provider, there is ^. good chance he or 
she will be referred to the proper youth agency. In short, competition 
for (job-ready) participants tends to be lower under market than function 
subcontracting. To the extent function subcontracting provides services 
less suited to participant needs, we might expect that services would be 
less effective under function than market subcontracting. Moreover, we 
would also expect that few participants would ever receive multiple 
services coordinated among multiple providers. However, this last 
feature might not be a problem, given PIC preferences for single 
component programs in the past. 

This pattern may not apply to SDAs subcontracting part of their services. 
For SDAs that subcontract training but retain control of outreach, 
intake, and placement services, the SDA still controls whether or not the 
participant enters a program as well as influencing tite type of training 
the participant receives. Conversely, subcontractors often have little 
influence over whom they train. SDA-subcontractor conflicts may arise 
over the types of participants referred for training. 

The second negative consequence is that, under function subcontracting, 
the SDA may loose the "big picture." That is, service delivery becomes 
fragmented when an SDA subcontracts to several organizations, each having 
multiple contracts to train participants for specific jobs. With service 
delivery dispersed in this manner, SDAs may have difficulty knowing where 
they stand performance-wise at any given moment. Here, the SDA must rely 
heavily on its MIS to monitor entry, training and outcome activities 
across providers on a regular and timely basis. Only the MIS can provide 
the SDA with an overview— the big picture— of where things stand for the 
SDA as a whole. 



EMPLOYMENT (STEP 7) 

At this point in the pathway, our focus is to exajTiine the outcomes 
associated with tue different types of services participants receive. 
Procedures in the gross impact guide (Volume IV in this series) may 
followed. Performance control systems based on an MIS are 'i-articularly 
suited for performing such analyses. While such analyses may be used to 
identify subc -tractors having problems with specific standards and to 
pinpoint where the problem exists in the pathway, they may say little 
about the underlying causes of the problen. For this, more thorough 
process analyses as discussed in this volume are needed. 

One point deserves mentioning here. As shrwn in the pathway, 
participants may exit into employment from all four competency areas. 
For example, a youth might want a job now and not have the patience to 
enroll in s training program. Thus, it might be highly informative to 
store the sequence of services in each participant's ITP in the MIS. 
Then, further analyses of how exit points in the sequence— start, middle 
or end— relate to outcomes may be conducted. Such information would be 
highly useful for staff responsible for designing the ITPs and may ^ead 
to important design improvements in this area. 

er|c - 



According to Exhibit 11, 98 placements occur, or about 80 percent of 
those enrolled, but only 45 percent of those who apply. 



FOLLOW-UP (STEPS 8 AND 9) 

Most of JTPA's performance standards reflect short-term outcomes, and so 
this is what SDAs and subcontractors have emphasized in their programs. 
Yet, as indicated in the hierarchy of objectives, positive terminations 
are only a means to an end, long-term job retention at a wage above the 
poverty level. Although most SDAs have limited follow-ups to 90 days (or 
less), PICs and states have shown some interest in conducting long-term 
retention assessments. Hov/ever, because JTPA services are one-shot, 
short-term interventions that alter only a few variables in the complex 
process determining labor market success, rea' limits may exist on what 
can be achieved in the long run. 

During follow-up, gross impacts should be monitored (to determine what 
the impacts are) and eval!iated (to determine the relative effects of JTPA 
services on the impacts). Methods for accomplishing this task are 
described the guide for gross impact evaluation. 

As indicated by Step 8 in the pathway in Exhibit 11, subcontrac^^ors (or 
the agency responsible for placement) often provide follow-up ser-ices to 
troubleshoot any problems that might arise, ultimately to assure that 
participants remain employed. Both employer and participant follow-up 
are commonly made. These services should be recorded in the MIS, and 
their relative contributions to long-term job retention should also be 
assessed through the gross impact evaluation. In our example SDA, 64 
participants, or 65 percent of the placements, are still employed one 
year following program completion. Their average wage and annual 
earnings are both above the poverty level. 



EMPLOYERS 

Up until now, our focus has been participants. But as they move through 
the pathway, SDAs and subcontractors are also performing job 
development — securing unsubsidized public and private job slots for their 
cl ients. These services are usual ly of two types (and most agencies 
usually do both): finding jobs for specific participants, and developing 
jobs available to all participants who wish to apply. As shown in the 
hierarchy of objectives (Exhibit 10), participants and employers usually 
connect with each other at distinct points in the hierarchy (or pathway). 
17/ Because these connections reflect coordination of SDA/subcontractor 
activity, thay are discussed in greater detail in the next chapter. 



No matter what services participants receive, they can conduct their 
own, independent search for employment concurrent with participation 
in JTPA. Sometimes participants are successful in finding a job. 
In these cases the employer-participant match can occur at any point 
in the hierarchy. 



ASSESSMENT OF ,riE WITHIN AND BETWEEN CONDITIONS 



For the most part, the work component in our SDA (and subcontractor) 
satisfies the WITHIN condition. The work component consists of 
well-defined, logically sequenced steps that move the participant 
relatively smoothly from entry to exit. A competency-based service 
system is in place to document that participants experience verifiable 
gains in skills, motivation arid other behavioral dimensions. In short, 
the work component is an integral element of the conversion process. 

The BETWEEN condition is satisfied too, but in a curious way. As you may 
recall from Chapter 3, the performance standards and goals of our SDA's 
mission component were inconsistent with the standards dominating the 
mission component. Some elements of the work component are consistent 
with the SDA's goals, while other elements are consistent with the SDA's 
performance standards. On the one hand, by attempting to provide most 
eligible clients with multi-component services targeted to meet their 
needs (as defined in the SDA's competancy-based system), the SDA has 
achieved its goal of implementing a comprehensive service delivery 
systsm. On the other hand, insufficient resources and pressures to meet 
performance standards push the SDA toward intake procedures and program 
structures that screen out les.9 motivated clients and that result in 
service creaming. Thus, a social consequence of these forces is that 
some eligible clients (such as the less motivated and, therefore, 
hard-to-serve) never are enrolled, while some enrolled participants 
receive something less than the treatments prescribed in their TTPs. The 
latter is one potential cause of the subcontractor's performance 
problems. 

Finally, we have waited to the very end to make a final point. All 
phases of the pathway— recruitment, intake, training, and so on— require 
qualified staff if the SDA or subcontractor is to perform well. Thus, 
whenever any phase of activity along the pathway is examined, so should 
the qualifications, skills and motivation of staff performing them. 
Ideally, staff competencies should match program characteristics of the 
organization. For example, an OJT subcontractor might hire staff with 
marketing skills for recruiting employers rather than staff with social 
work and client case management skills. 



94 



CHAPTER 5 

HOW TO EXAMINE THE COORDiNATiON COMPONENT 

Overview of the Chapter 

Definition of the WITHIN Condition 

Definition of the BEr f/EEN Condition 

The WITHIN Oo^idit^on 

Background: Five Different Ways of Coordinating Activity: 
•Standardize Outputs 
•Standardize Skills 
•Standardize Tasks 
•Mutual Adjustment 
•Direct Supervision 

Coordination in the Mission Component 

Coordination in the Work Component 
•Intake 
•Training 
•JTPA Connection 
•Follow-up 
•Summary 

The BETWEEN Condition: Coordinating Activity in the Mission 
Component with Activity in the Woric Component 

The BETWEEN Condition: Coordinating Activity Between 
Organizations— Some Critical Issues 

Accountability 
Comprehensive Services 
Intake and Training 
. .3DA— Community Agencies 
Employer Services 
Goal Distraction 
Summary 

Assessment of the WITHIN and BETWEEN Conditions: An Example 
Q Discussion 

ERIC ' 



CHAPTER 5. HOW TO EXAMINE THE COORDINATION COMPONENT 



In the work component, our attention focused on SDA and subcontractor 
treatments (or the service strategy) provided to participants. Elements 
of the work component primarily determine the organization's effective - 
ness , in producing outputs—in this case, increased employment and 
earnings and reduced welfare dependency. The coordination component, in 
contrast, captures the organization's efficiency in producing these 
outputs. Common indicators of an organization's efficiency include cost 
per placement, cost per training session, average time to complete an ITP 
and others. 

Our aim in this chapter is to present guidelines for examining the 
coordination component, or the organizational processes that determine 
its efficiency. The guidelines are presented in three sections. First, 
guidelines are described to determine whether the coordination component 
satisfies the WITHIN condition. For the coordination component, the 
WITHIN condition requires that the mission component and the work 
component have machanisms for coordinating their respective activities. 
In the mission component, for example, management is a major coordinating 
mechanism; it assures that the parts of the pV^nnlng process— choosing 
goals, making allocation decisions, etc. --are coasi .ent with each other. 

The second section presents guidelines for examining the BETWEEN 
condition. For the coordination component, the BETWEEN condition 
requires that mechanisms exist (1) to coordinate activity in the work 
component with activity in the mission component, and (2) to coordinate 
one organization's activity with other organizations in the environment 
(for example, mechanisms for coordinating an SDA's activity with its 
subcontractors and outside agencies, such as welfare). 

The third section illustrates thtse guidelines in the example SDA that we 
have been examining. 



COORDINATION MECHANISMS 

Every organized human activity has two fundamental and complementary 
requirements: the division of labor into various tasks to be performed, 
and the coordination of these tasks to accomplish the activity. The 
structure of an organization is how labor is divided into distinct tasks, 
and how coordination is achieved among the tasks. [17] 

There are several ways to coordinate activities in organizations, but 
most involve various forms of standardization. [16, 17] SDAs and sub- 
contractors coordinate service delivery in five ways. First, co- 
ordi nati on i s achi eved standards z i ng outputs through performance 
standards » SDA activity is directed, or coordinated, at achieving the 
standards, but how each SDA does this often varies. 

The second method of coordination is standardizing skills. This means 
that SDAs hire trained staff having simi'^ar--or standardized— employment 
and training skills. But no matter how standardized the knowledge and 
skills of staff, their complexity ensures that considerable discretion 
remains in their application. No two staff ever apply their skills in 
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exactly the same way. Each staff member works fairly independently of 
others but closely with the clients he or she serves. Yet, because 
skills are standardized, each staff member knows what to expect from 
others, and so coordination can be achieved. In short, SDAs coordinate 
activity by standardizing the worker as well as the outputs of the 
service delivery systems. Yet, this is only a loose coordination 
mechanism; perfect coordination of an organization's activities is rarely 
assured through this mechanism. 

Third, organizations also coordinate activity by standardizing the task 
to be performed. For example, a youth provider may have a subcontract 
requiring a high volume, low cost per placement operation. The provider 
judges that most youth that it serves have very similar needs. There-- 
fore, most youth can receive the same set of quick-turnaround services. 
In other words, because the needs of youth are similar, the provider can 
standardize the servTce(s) that each participant receives. All youth 
coming throi.qh the agency receive the same service(s), and service 
delivery beco.nes highly repetitive. Under these conditions, staff skills 
become less imporUnt. The provider only needs staff that know (or can 
quickly learn) a su.<i]] number of skills and then consistently apply these 
skills to all youth. In this way, standardization of the task co- 
ordinates the provider's operations. Compared to coordinating by staff 
skill, this form of coordination is more formalized, less flexible and 
more bureaucratic. And it may require additional staff simply to co- 
ordinate work flows. Yet it is the organizational structure best suited 
for efficiently placing a high volume of participants (assuming services 
are effective). 

The fourth method of coordination, mutual adjustment, is simply informal 
communication among those doing a task. For example, two persons 
paddling a canoe rely on mutual adjustment to coordinate their efforts. 
Similarly, JTPA intake staff may discuss as a group what services each 
client should receive. 

The f i fth method is direct supervision, where one person takes 
responsibility for the work of others, issuing instructions to them and 
monitoring their actions. In effect, one brain is coordinating several 
hands. In JTPA, examples are the management (or coordination) of an 
organization's intake, planning or other unit's activities. 

In summary, an organizat can coordinate its activity by standardizing 
outputs, staff skills, and tasks, as well as through mutual adjustment 
and direct supervision. As a whole, the three modes of standardization 
also aid management control of the organization. We briefly review an 
organization's coordination mechanisms in the mission and work components 
below. 



THE WITHIN CONDITION: EXAMINING THE MISSION COMPONENT 

The mission component has two chief means of coordinating its activities, 
staff skills rtnd managements The various tasks of the mission 
component — developing goals and objectives, conducting a planning process 
and operating the performance control system — all require competent staff 
with relevant skills, if the tasks are to be performed correctly. For 




example, an SDA may employ three planners, each having a different 
planning task to perform. If one of the planners lacks skills to perform 
her or his task, little coordination among the planners may occur, and a 
low quality plan may result. 

Management also plays a coordinating role in the mission component, ^'fy-r 
exampl e , in the above pi anni ng exampl e , management can per ; ^ ' y 
review the products of each planner to assure they fit together^- • ly 
into a future job training plan. A second example is management : > 
in coordinating the SDA's planning and compliance efforts. In many v 
planning and compliance are independent activities. Planners develop 
plan; someone else is responsible for implementing it and monitor 
performance. Consequently, planners may know little about how well 
plan is working. If performance problems are detected, planners, who 
often have the greatest understanding of local conditions among SDA 
staff, are often "left-out" of the SDA's problem solving efforts. 
Management, however, can bring planners and compliance staff together 
(when appropriate) to help resolve such issues. 



THE WITHIN CONDITION: EXAMINING THE WORK COMPONENT 

All five ways of coordinating activity are commonly found in the work 
components of JTPA organizations. In the sections that follow, co- 
ordination issues in the client pathway are discussed. 

INTAKE . No matter what organization performs intake, a group of staff 
must work togethfir to move clients through the appl ication-assessment- 
assignment process as efficiently as possible. Common forms of co- 
ordination here are standardization of skills and mutual exchange of 
information among staff. For example, staff may have very similar 
training, and participants are assigned services based on discussions 
among all staff. Therefore, if intake staff have deficient skills (or do 
not share common skills), or if staff do not talk to each other (because 
they do not like each other), intake may run poorly. These problems can 
be solved through staff training, shifting staff assignments, replacing 
staff, or increased supervision. 

TRAINING . JTPA organizations sometimes develop multi-service ITPs for 
participants. Where this occurs, mechanisms must exist for coordinating 
service delivery— to assure that each participant moves smoothly from one 
service into another in his or her plan. A common mechanism is assigning 
cl ients to staff counselors , who are resonsible for monitoring the 
client's progress through his or her ITP. To increase staff pro- 
ductivity, some agencies may store the client's ITP on the MIS. Then, 
the MIS automatically prompts staff when each client is scheduled to move 
to his or her next service. 

Extensive coordination also normally occurs during training itself. The 
direct supervision provided in an individual's OJT program or instructor 
supervision of a vocational education class are also examples of this 
type of coordination. 

Agencies operating competency-based systems also have additional co- 
ordination concerns. Not only must services be delivered properly but 
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each participant must also progress in a predetermined, step-by-step 
manner through the various competency levels. The chief concern here is 
coordinating services with each participant's incremental achievements. 

JTPA CONNECTION. At some point along the pathway the participant is 
matched with the employer (or some other positive outcome occurs). 
Although some participants find employment on their own, many rely on 
JTPA staff and services to perform tlie connection. There are several 
ways of coordinating the matching process. One is to have staff be 
responsible for performing both activities; staff develop slots as needed 
for their respective participants. The potential problem with this 
approach, however, is that one staff member may develop slots best suited 
for participants assigned to other staff. A solution to this problem is 
to make participant assignments a group activity. Staff would meet 
regularly as a group, where all the participants and slots are brought 
together and appropriate assignments made. When staff have similar 
skills and criteria for matching, this format is a relatively direct 
method for handling an otherwise cumbersome task. Finally, if separate 
staff are responsible for training and slot development, another solution 
IS to make each unit's manager responsible for both activities, or have a 
separate staff member coordinate work in the two units. The essential 
point here is that SDAs must have workable mechanisms for connecting 
participants with training and job slots. 

Exhibit 10 in Chapter 4 defines locations in the hierarchy where 
participants can be matched with employers. The following analyses may 
be conducted to determine how well the participant-employer matching 
process is performing in your SDA or subcontract organization. 

First, the most frequently used procedures (or services) for matching 
participants with employers are listed and defined. This simple exercise 
is conducted for two reasons. First, it allows agencies to document the 
scope of their matching activity. Some organizations are surprised to 
see how few services are devoted to accomplishing this function. Second, 
by defining the service, some agencies doing job development may discover 
that very little staff effort goes into the service. 

Once the services are listed, the second step is to find out what percent 
of enrol lees received them. This answers the question, "What percentage 
of enrol lees found jobs on their own (through job clubs, JSA, etc.) 
versus were referred to jobs located by staff?" 18/ The underlying issue 
here is control of the matching process. If the majority of participants 
find jobs on their own, the agency's placement performance is determined 



18/ The percentage of participants finding jobs on their own is 
influenced by the agency's service mix. For example, an agency with 
high numbers of OJT participants may have a low percentage of 
participants finding jobs on their own. Conversely, agencies with a 
high number of institutional training participants may have a 
majority finding jobs on their own. 
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mainly by participants. On the other hand, if the majority of 
participants are placed through job development services, staff have 
greater control over the participant-employer matching process. Thus, if 
the agency is having a placement problem, it may ba caused by how 
participants find jobs. If placement rates are too low and most partic- 
ipants find jobs on their own, the agency may wish to consider placing 
more participants through job development (to increase agency control 
over the matching process). But if mst participants are placed through 
staff job development, yet a placement problem exists, the problem may 
lie with the quality of the service, or perhaps a poor job market. 

Agencies should repeat this exercise for employers. For example, 
employers hiring participants through job development or referral and OJT 
services may be compared with employers hiring participants through JSA, 
job clubs and related services. How do their satisfaction levels differ 
(if at all)? Is either group a "regular customer^*--employers who 
repeatedly hire JTPA participants? 

The third step is to define where in the pathway most matches occur. Do 
most occur at (or shortly after) termination? Or at intake (when 
participants are assigned to OJT slots)? This type of analysis is useful 
in assuring that the timing of job development/placement services is 
appropriate for most participants (and employers). 

These are some basic analyses that may be conducted of the matching 
process. The chief tool for conducting these analyses is a properly 
designed MIS containing both participant and employer data. You may wish 
to expand or customize these analyses to meet the unique requirements of 
your own agency. In doing so, please be aware that when examining the 
matching process, every question that you ask of participants can usually 
also be asked of employers. 

FOLLOW-UP . After the participant completes training and exits the 
organization, staff conduct one or more follow-up interviews with the 
participant and (sometimes) his or her employer. Services may also be 
providad during this stage. All of this activity must be coordinated: 
the task must be integrated into staff work schedules, telephone calls 
must be scheduled at proper dates following termination, and any follow- 
up services must be arranged. Although management can play a big role in 
coordinating these tasks, the MIS can relieve some of this burden and 
thereby make staff more productive. When programmed properly, the MIS 
can print weekly reports listing all follow-up participants to be 
contacted that week, along with their telephone numbers and the numbers 
of their employer. Then, management's only task is to make sure the work 
gets done. 

SUMMARY . In any JTPA organization the efficiency of the flow of clients 
along the pathway is greatly determined by how well the organization 
coordinates its activities. 

Process indicators often used to measure flow efficiency along the 
pathway are workflow, volume and productivity [1:56]: 

• Workflow can be measured in terms of cl ient time (e.g., 
average time between call for appointment and the appoint- 
ment , between begi nni ng and compl eti on of i ntake , between 
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intake and admission, between admission and discharge or 
termination) and/or worker time (e.g., average time to 
complete writeup of intake, average time to complete 
di scharge pi anni ng , average time to compl ete job skills 
education program, average time to complete crisis 
referral s). 

• Volume can be measured in terms o*^ number of cl ients served 
(e.g. , weekly, monthly, annual ly) and/or the number of 
services provided (interviews conducted, counseling 
sessions, etc.). Chapter 4 contains an illustration of how 
client volumes along the pathway can be assessed. 

• Productivity can be measured in terms of performance ratios 
related to costs, staff, and time . Costs can be related to 
units of service or activities (e.g., cost per counseling 
session, classroom training session, diagnostic assessment, 
discharge plan^^ing, etc.). Staff ratios can be related to 
workload standards (number of cases per counselor, number of 
participants per class, number of community information 
meetings per worker, etc.). Time ratios can be related to 
the frequency of a program activity in a given time frame 
(e.g., number of clients screened for eligibility per month, 
number of counseling sessions per week, number of training 
or activity sessions per day). 

While SDAs and subcontractors should periodically use these indicators to 
monitor the efficiency of their respective organizations, having an 
efficient organization does not guarantee a successful program. This is 
because efficiency is just one of several factors determining perfor- 
mance. For example, two subcontractors with identical expenditure/ 
personnel, expenditure/ participant, and participant/personnel ratios may 
differ substantially in performance, depending on hovy decisions are made 
about specific aspects of treatment. [11] In the next section, other 
factors inf luencirig the efficiency of the organization are discussed. 



THE BETWEEN CONDITION 

The BETWEEN condition requires that mechanisms exist (1) to coordinate 
activity in the work component with activity in the mission components; 
and (2) to coordinate an organization's activity with other organizations 
in the environment. These are reviewed below. 
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COORDINATION BETWEEN MISSION AND WORK COMPONENTS 

In the mission component, access, treatment and allocative decisions 
determined what groups would receive services, how many individuals would 
be served, and how much the total costs of each service would be. In 
other words, a plan was developed for meeting the SDA's goals (and 
performance standards). 

Once implementation begins, however, intake activity may not follow the 
SDA's plan. Applicants may have unexpected needs and earning or 
employment barriers. Some target groups may not apply in sufficient 
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numbers to meet performance standards. Applicants may not meet employer 
needs. In short, implementation does not follow the plan. 

In these cases, it is management's role to make sure intake coordinates 
its activity with on-going access, al locative and treatment decisions (of 
the mission component). That is, intake must receive timely, regular 
information from the performance control system, such as the current 
number of enrol lees, services assigned-to-date, expenditures-to-date, and 
progress toward meeting performance standards. If management detects, 
for example, that OJT slots are being assigned too rapidly, management 
may direct intake to alter its OJT assignment procedures (at least until 
the problem corrects itself) or take other corrective actions. Without 
such coordination of intake with the mission component, there is little 
chance that SDAs (or contractors) will meet their performance require- 
ments. Similar forms of feedback should also occur at the treatment and 
follow-up stages of the client pathway. 



COORDINATION WITH OTHER ORGANIZATIONS 

If an S?A decides either to subcontract services or enter coordination 
agreements with outside agencies, mechanisms must exist for coordinating 
the SDA's activity with the other organizations. Normally, the following 
coordination issues arise in these arrangements. 

ACCOUNTABILITY . Because coordination is rarely perfect, lines of 
accountability become unclear, particularly for SDA's subcontracting all 
services by function. When services are contracted by function across 
providers, accountability for achieving SDA performance standards cannot 
be easily measured or determined. If placements (or another standard(s)) 
fall, who is at fault? Is it with a specific provider? Or with 
contracts for a specific service? Or does the problem lie with the SPA? 
Has the SDA's planning process gone awry, targeting resources and 
contracts for jobs that do not exist in the labor market? There may be a 
lot of fingers pointed, but determining actual causes of poor performance 
is difficult at best under this form of service del ivery. However, 
accountability tends to be preserved in SDAs subcontracting by market. 
If only one agency is providing youth services, then it is clear who is 
accountable for youth performance. And if it can be determined who is 
accountable for a given coordination problem, developing new SDA-provider 
coordination mechanisms to solve the problem becomes much easier. 

Similar accountability issues arise for SDAs that subcontract only part 
of their services. Here, accountability is shared by the SDA and the 
subcontractor. Who is at fault when performance falls? Does the problem 
lie with the SDA's intake and assignment procedures, or with the sub- 
contractors' training programs? A careful, impartial analysis of both 
operations (and their interrelationships) is needed to untangle and 
clarify who is accountable in these service arrangements. 

COMPREHENSIVE SERVICES . If an SDA offers comprehenswe sei-vices from 
multiple subcontractors, mechanisms must be present to assure that 
participants move from one service to another in the proper sequence, as 
stated i n the ITP . A common mechani sm i s ass i gni ng c 1 i ents to staff 
counselors, who are responsible for monitoring the clients' progress 
through the ITP. To reduce staff effort required here (which may be 
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large, depending on staff-client ratios), the MIS can automatically 
prompt staff when p:)ch client is scheduled to move to his or her next 
service (provided the ITP is stored in the MIS). 

Such monitoring tools are important when an SDA (unwisely) offers 
comprehensive services through function subconv/acting. Here, different 
subcontractors are responsible for each service in the ITP, and only 
through direct staff efforts can services be coordinated properly. This 
method of providing comprehensive services is inefficient because it 
involves a great deal of staff time to coordinate (and, therefore, higher 
administrative costs). 

INTAKE AND TRAINING . If the SDA performs intake and contractors perform 
training, the two organizations must coordinate their activities. This 
involves two major issues. First, SDAs with this service arrangement 
must assure that the flow of participants from the SDA to the contractor 
operates in a smooth and timely manner. The MIS should be used to 
calculate the average number of days between enrollment and first day-of- 
service as an indicator of pathway efficiency. The smaller the average, 
the more efficient the flow of participants. 

Second, SDAs and contractors must share similar "validity assumptions" 
(Exhibit 10 in Chapter 4) for the transfer of participants to work. That 
is, both parties must define the participants' needs and barriers in a 
similar way, and both must agree that the services provided by the 
contractor are appropriate for the individual. Otherwise, contractors 
may become highly dissatisfied with SDA referrals, resulting in poor 
SDA-contractor relations. Just as intake staff rely on standardization 
of skills and mutual exchange of information to coordinate their efforts, 
so must the SDA and contractors use these mechanisms to coordinate their 
efforts. 

SDA-COMMUNITY AGENCIES . The SDA must also coordinate its services with 
other agencies in the area, such as welfare. Such effort is required by 
the JTPA to increase services available to participants and to satisfy 
JTPA's welfare performance requirements. Thus, our interest is in 
defining actual, working client referral relations between the SDA or 
subconuractor and the outside agency. If an SDA or subcontractor has 
established relations with outside agencies but no referrals are being 
made, then this form of service coordination is a failure and may 
contribute to poor performance. 

The MIS niay be used to produce the following descriptive statistics, 
which profile referral activity: 

• Total number of clients enrolled; 

• Percent of enrol lees referred from outside agencies; 

• Percent of services that referred clients receive (from JTPA 
and from all outside sources); and, 

• Outcomes of referred enrol lees. 




These statistics may be calculated for all enrollees (as listed above) or 
separately for each outside agency. They measure the scope of referral 
activity and, if found deficient, may help management make informed 
decisions for improving referral relations. For example, one multicounty 
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SDA was having problems meeting its welfare performance standard. It 
soon discovered that virtually all of its welfare referrals were from a 
single county, where the only welfare office in the SDA was located. 
Recruitment was intensified in other counties, and SDA-welfare relations 
were modified to correct the situation. 

SDAs should also check for possible coordination problems between 
subcontractors and outside agencies (such as welfare). Each sub- 
contractor may have to establish links with outside agencies. The likely 
result is that outside agencies may be overwhelmed by maintaining so many 
links, producing fragmented coordination patterns for the SDA as a whole. 

EMPLOYER SERVICES . In a similar way, each subcontractor is responsible 
for employer services, and employers may be bombarded by agencies, all 
seeking placements, OJT slots, or other forms of participation in their 
respective^ programs. This kind of overkill can reduce employer 
participation in an SDA*s programs and, where severe, pressure the SDA to 
take over this activity. 

GOAL DISTRACTION . Subcontracti ng can di stract SDAs from thei r 
goals. [16, 17] Once the SDA has decided to subcontract, attention 
usually shifts to establishing and monitoring contracts, and the SDA's 
goals may become lost (particularly if the SDA*s goals are not clearly 
defined). The result is often an "ends-means" reversal, where the SDA's 
mission shifts from achieving a set of goals to administering a set of 
contracts. To keep itself on course, the interrelationships of contracts 
to performance standards and performance standards to goals must be 
well-defined and kept in mind in the day-to-day operations of the SDA. 

SUMMARY . These points indicate that subcontracting has its pluses and 
minuses. More centrally, subcontracting involves a fundamental tradeoff 
in the use of resources. [16, 17] By subcontracting (and, hence, service 
specialization), SDAs gain as resources are used more effectively. But 
these gains can be offset by the need for more personnel to coordinate 
services (i.e., a potential loss of efficiency). Subcontracting is 
efficient only when it saves sufficient resources to pay at least the 
higher costs of increased management coordination. 



ASSESSMENT OF THE BETWEEN AND WITHIN CONDITIONS 

We will usa our SDA problem to illustrate how the BETWEEN and WITHIN 
conditions are applied. An examination of our example SDA reveals that 
intake staff are working well as a unit and closely coordinate their work 
with the performance control system; well-established links with non-JTPA 
agencies in the community exist; but SDA and subcontractor staff have 
voiced problems with the JTPA connection and SDA/subcontractor relations. 

Whenever an SDA subcontracts a portion of its services, it is essentially 
lifting a chunk out of the pathway and giving it to someone else to 
perform. While service delivery may be more effective through this 
allocation of effort, SDA and subcontractor activities must now be co- 
ordinated. Discussions with SDA and subcontractor staff across a variety 
of positions (including management) reveals the following symptoms of 
coordination failure. 
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• SDA staff perform intake using their competency- based 
system. The subcontractor operates training programs under 
a somewhat different needs-based system. Consequently, the 
SDA and subcontractor disagree on how services should be 
assigned, and the subcontractor is often dissatisfied with 
the participants the SDA refers for training. 

• The SDA's competency-based system requires SDA intake staff 
to monitor participant progress through training. The 
subcontractor regards thi s as unneeded compl i ance/ 
harassment. Monitoring occurs only when an SDA staff member 
has a good working relationshTp with someone in the sub- 
contractor's organization. 

• The subcontractor is responsible for placing the 
participant, yet it is SDA intake staff who are most 
familiar with his or her employment plans, competencies, 
etc. 

• The subcontractor is supposed to refer participants with 
multi-component ITPs back to the SDA when the sub- 
contractor' s service is completed. Sometimes this never 
happens. 

• Performance standards are not being met, even though 
everyone seems to be working at their full capacity, and 
intake and job training services, by themselves, seem to be 
wel 1 -designed and executed. 

This evidence leads to the following conclusions regarding the WITHIN and 
BETWEEN conditions for the coordination component. 

• WITHIN CONDITION: This condition is satisfied in part. The 
SDA and thcj subcontractor appear to have well-designed 
mission and work components with few coordination problems 
in each. However, SDA and subcontractor staff are dis- 
satisfied with how the JTPA connection (i.e., the process of 
matching participants with employers) is operating in the 
work component. The source of this dissatisfaction is found 
in the BETWEEN condition. 

• BETWEEN CONDITION— MISSIONA^ORK COORDINATION: Few problems 
exist here, except that subcontractor staff (work component) 
view SDA monitoring as unjustified. This is a ijymptom of 
the true causes of the problem, which are explained next. 

• BETWEEN CONDITION— SDA/AGENCY COORDINATION: The SDA has 
establ ished good working relationships with non-JTPA 
agencies. However, coordination between the SDA and sub- 
contractor is a failure. The SDA and the subcontractor have 
different pigeonhol ing processes for assigning services. 
Consequently, SDA intake decisions are often inconsistent 
with the subcontractor's treatments. Also, no mechanism 
exists to coordinate SDA-subcontractor referrals. Further, 
no mechanisms exist to coordinate participants' progress 
through their respective ITPs. 
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In short, the SDA and the subcontractor have failed to coordinate their 
activities. In this particular case, both the SDA and the subcontractor 
are accountable for the subcontractor's recent poor performance. 

l^^^ example shows how to use the guidelines for examining coordination 
•s.' . ^^^^^^^^^^^^^^^^ SDA and a single agency. For a comprehensive assessment of 

SDA- subcontractor relations, the guidelines must be applied to each of 
SDA's major subcontractors. 



io-*v;^f^|c.;;.. .-;yv,. CONCLUSION 

- ■ The material presented in this chapter and the previous one (on the Work 

Component) reinforce how important staff skills are to the success of the 
SDA. The work component relies; on staff skills for effective services, 
^•^i^fes.n-'; =..Vv: T?»e. coordination component relies (in part) on staff skills for efficient 
^P^S^f - '4.^K^0Ptf;»P^ delivfiry (that is, the proper coordination of activity). Without 
W^¥'ti--'i-^-^'-W quality staff, .performance is bound to suffer. This under- 
scores the critical importance of personnel decisions , or who is hired in 
the organization. Hiring and retaining qualified staff is a key 
, . ingredient to running a successful program. And one study of CETA found 

■l^^%vlr-.: "-= '. •'^'^f* attracting and retaining competent and dedicated staff was directly 
' if'^iated to the strength of leadership and having a well-defined sense of 
"'■'ssion. {93 So, now we see how parts of the organization are inter- 
related: strong governance (leadership) results in a well-defined 
mission, in turn attracts quality staff for the work and 

coordination components, which in the end results in the achievement of 
the organization's goals and performance standards. 
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CHAPTER 6. HOW TO EXAMINE THE SOCIAL COMPONENT 



Unlike the previous three components, the social component has neither 
WITHIN nor BETWEEN conditions. Instead, as described in Chapter 1, this 
component captures the social consequences of coordinating work to 
achieve the organization's mission. That is, in the day-to-day activity 
of the organization, strain, tensions, and conflicts inevitably occur 
aniong the members of the organization. Where severe, these conditions 
can undermine the organization's effectiveness and efficiency and 
possibly threaten its survival over the long-run. 

In this chapter our aim is to examine social conditions in the 
organization, which can be either the SDA or the subcontractor. First, 
guidelines are presented for examining staff, employer and participant 
satisfaction in the organization or service delivery area. Second, 
simple methods are presented for examining participant and employer 
satisfaction with the JTPA connection. In principle, the mission work 
and coordination components probably have few adverse consequences if 
most employers and participants are satisfied with how the connection is 
working. 

Finally, a brief discussion of values and leadership in the organization 
is presented. As in the previous chapters, we will apply these guide- 
lines to our example SDA problem. 



STAFF SATISFACTION 

In examining staff satisfaction, we are mainly concerned with the twin 
issues of staff retention and turnover. Knowing the importance of its 
staff, an organization may concentrate on minimizing staff turnover. 
This would be a mistake for two reasons. [8] First, there is only a 
loose connection between staff satisfaction and turnover. Some dis- 
satisfied employees never leave for a variety of reasons (e.g., a tight 
job market, financial responsibilities, etc. ). Second^ the reasons why 
staff stay are equally or more important for the organization that seeks 
to maintain a motivated and productive workforce. 

The main reason employees stay is inertia; employees tend to stay until 
some force causes them to leave. What factors affect this inertia? 
There are two relevant factors ^ inside the organization and also two 
factors outside the organization. 

First, within the organization, there is the issue of job satisfaction. 
Second, there is the "organizational environment" and the degree of 
comfort an individual feels within it. The more an employee's work ethic 
is compatible with the values for which the organization stands, the more 
likely the employee will stay. For example, in organizations oriented 
around the needs of employers, staff dedicated to training the hard-to- 
serve may eventually decide to leave the agency. 

)' 

Performance standards are perh'^s the most important source of stress 
(and, hence, potential dissatisfaction) among SDA and subcontractor 
staff. For SDA staff, operating the performance control system can be 
highly stressful. There are no hard and fast rules on when compliance is 
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needed and when technical assistance might be more appropriate. It is 
more of an art than a science. So, contract staff may have personal 
conflicts about what to do in specific situations. For subcontractor 
staff, performance standards exert pressures on staff to move partic- 
ipants into jobs as quickly as possible. Yet, these same staff are often 
de;1icated to helping participants with their employment and training 
problems. These two forces run counter to each other, and the disparity 
bet^ween the two may result in high levels of staff dissatisfaction and 
turnover. 

Outside the organization (either SDA or subcontractor), there is the 
employee's perceived job opportunities, plus nonwork factors, such as 
family ties, friendships, community relations and financial 
responsibilities. 

These considerations yield four prof i les of staff that are useful in 
thinking about the twin problems of employee retention and turnover. 
These are as follows. 

• The turn-overs : those who are dissatisfied with their jobs, 

have few environmental pressures to keep them in the 
SDA, and will leave at the first opportunity. 

• The turn-offs : employees who are highly dissatisfied with 

their jobs but stay mainly for environmental reasons. 
These can generate employee relations and productivity 
problems. 

• The turn-ons : staff who are highly motivated and remain 

with the SDA almost excl i:sively because they enjoy 
thei r work. They real ly want to stay and are not 
locked in by the outside environment. However, if 
managerial actions reduce job satisfaction, turnover 
may rise in this group. 

• The turn-on plus : those most likely to stay with SDA over 

the long-run. They stay for job satisfaction plus 
social and personal reasons. So, if job satisfaction 
temporarily declines, they will probably stay. But if 
sati sf acti on drops permanently these empl oyees may 
become turn-offs , i ncreas i ng f rustrati on and reduci ng 
productivity. 

In summary, people leave and stay with SDAs (and subcontractors as well) 
for complex reasons, and these reasons often vary by position, amount of 
training and other factors. Finding out why people leave the SDA will 
continue to be important for assessing job satisfaction. But if the SDA 
is interested in retaining quality staff (which are vital for a well-run 
organization), equal emphasis should be placed on managing retention of 
those one wishes to retain. This means discovering the reasons why 
motivated employees stay with the SDA and then reinforcing these reasons 
during day-to-day management. Th![is ^xil timately means that manipulative 
and conformist philosophies of management may contribute little to staff 
retention. Instead, management is needed that respects and reinforces 
staff abilities, values and interests— as these vary across positions. 



Because SDA staffs are usually small (relative to CETA), management may 
be in a better position to nurture a satisfying working environment by 
maintaining close staff-management relations and open communication. 



EMPLOYER SATISFACTION 

Assessing the extent of employer satisfaction with the JTPA program seems 
critical. Dissatisfied employers may not participate again (causing a 
loss of future slots), and they may influence other employers not to 
participate or to boycott the SDA. Here, our interest is to present 
guidelines for evaluating the extent of employer satisfaction either for 
the SDA as a whole or for a particular subcontractor. 

Employers should be asked detailed questions about the program to 
determine their satisfaction levels. These questions generally fall into 
two categories: (1) questions about JTPA services and programs as a 
whole; and (2) questions about the employer's satisfaction with a 
specific participant following placement. The first category may be 
asked periodical ly» say once a year, of employers participating in the 
program. The questions below are representative of the category. SDAs 
or subcontractors may wish to modify these to conform with their own 
needs and requirements. 
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EXHIBIT 12 
SAMPLE EMPLOYER QUESTIONNAIRE 



Q-1 In general, considering all your contacts with the SDA and all the 
participants you have hired, how satisfied do you feel with the SDA? 
(Circle the number of your answer.) 

1 EXTREMELY DISSATISFIED 

2 SOMEWHAT DISSATISFIED 

3 NEITHER SATISFIED NOR DISSATISFIED 

4 SOMEWHAT SATISFIED 

5 EXTREMELY SATISFIED 

Q-2 If you have a question about our programs or someone you hired, do 
you know who to contact in our agency? 

1 YES , 

2 NO / 

3 NOT SURE 

Q-3 Listed below are some items describing our employer services. For 
each item, please indicate if you are SATISFIED, DISSATISFIED, or 
NOT SURE. 



CIRCLE YOUR ANSWER 

SOMEWHAT VERY 
VERY SOMEWHAT DIS- DIS- 



ITEM 


SATISFIED 


SATISFIED 


SATISFIED 


SATISFIED 


A. 


Amount of paperwork required 


v-s 


s-s 


S-D 


V-D 


B. 


Amount of time to fill job(s) 


v-s 


s-s 


S-D 


V-D 


C. 


Dealings with SDA staff 


V-S 


s-s 


S-D 


V-D 


D. 


Dealings with subcontractor staff 


V-S 


S-S 


S-D 


V-D 


E. 


Applicants referred by program 


V-S 


S-S 


S-D 


V-D 


F. 


Information describing program 


V-S 


S-S 


S-D 


V-D 


G. 


Number of contacts with staff 


V-S 


S-S 


S-D 


V-D 


H. 


Timely reimbursement of wages 
(if applicable) 


V-S 


S-S 


S-D 


V-D 
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Q-4 Can you think of any other services that might help you and other 
employers in our programs? 

1 NO 

2 YES J 

(If YES, please describe): 



Q-5 Are staff attentive to the particular employment and training needs 
of your business? 

1 ALWAYS 

2 SOMETIMES 

3 NEVER 

Q-6 Based on your past experience, how likely do you think your business 
will hire another applicant from our program in the future? 



1 VERY LIKELY 

2 SOMEWHAT LIKELY 

3 SOMEWHAT UNLIKELY 

4 VERY UNLIKELY 

The second category consists of questions about the employer's 
satisfaction with a specific participant following placement (or in an 
OJT program). To minimize paperwork, these questions should be few in 
number. The following list is representative and may be modified to suit 
your SDA s circumstances. [28] For example, in a competency-based 
system, the list of traits in Question 3 (below) may be identical to the 
indicators used to measure participant competencies. 

Q-1 Does the employee still work for you? 



Q-2 In general, how satisfied are you with the employee? (Circle the 
number of your response. ) 



1 
2 



NO 
YES 



STOP 



1 
2 
3 
4 



VERY SATISFIED 
SOMEWHAT SATISFIED 
SOMEWHAT DISSATISFIED 
VERY DISSATISFIED 
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Q-3 



Following is a list of employee traits. For each trait, please 
indicate if you are VERY SATISFIED, SOMEWHAT SATISFIED, SOMEWHAT 
DISSATISFIED, or VERY DISSATISFIED with the employee. 



(Circle your answer) 



SOMEWHAT VERY 

VERY SOMEWHAT DIS- DIS- 

SATISFIED SATISFIED SATISF IED SATISFIED 

TRAITS ' 

A. Works well with hands V-S S-S S-D V-D 

B. Neat, attractive appearance ''-S S-S S-D V-D 

C. Always on time for work V-S S-S S-D V-D 

D. Willing to do extra work V-S S-S S-D V-D 

E. Willing and able to follow 

orders V-S S-S S-D V-D 

F. Works well without waiting 

for instruction V-S S-S S-D V-D 

G. Easy for others to get 

along with V-S S-S S-D V-D 

H. Intelligent V-S S-S S-D V-D 

I. Enthusiastic V-S S-S S-D V-D 
J. Honest and trustworthy V-S S-S S-D V-D 
K. Able to work fast V-S S-S S-D V-D 



If your MIS contai ns a master file of empl oyers and descriptive 
characteristics (such as number of employees, type of firm, etc.), 
several types of analyses may be performed with these data. For 
questions about the SDA's programs, frequency distributions of the 
responses from al 1 employers may be computed to provide an overal 1 
response profile for the SDA. Separate distributions may be generated 
for di f f erent categori es of empl oyers , such as retai 1 , manuf acturi ng , 
professional, etc., to determine whether satisfaction levels vary across 
categories. The data may also be examined by the services employers 
received. For example, satisfaction levels of OJT employers may be 
compared with levels among employers receiving only placement services. 

If these comparisons show that most groups of employers are satisfied 
with the SDA's programs, it would suggest that quality is consistently 
hi gh across programs . If di f f erences are detected , however , i t may 
indicate that not all employers, are satisfied with the program. By 
isolating program areas where employers are dissatisfied, SDA (sub- 
contractor) staff may target their efforts to improve program quality in 



these areas. Regularly monitoring employer satisfaction over time is one 
way of checking whether such improvements have increased satisfaction 
levels. Given that most SDAs operate programs oriented around employer 
needs, this feedback information is helpful in maintaining employer 
interest and participation in JTPA over the years. 



PARTICIPANT SATISFACTION 

In general^ participant satisfaction reflects the qual i ty of services in 
the SDA. Other thi ngs bei ng equal , we mi ght expect sati sf acti on to 
increase as the quality of services goes up. Unfortunately, this is 
difficult to discover in practice. If you ask the average American 
whether he^ or she is satisfied with something, the typical answer is 
"YES." While this might be a comforting response, it does not tell you 
very much. Maybe it tells you that you are doing something right, but it 
does not help you improve the quality of your services. Only by becoming 
more^ detailed— by asking participants, employees, and staff about 
specific aspects of the program—can the good and bad parts of the 
program be identified. 

A previous study of participant satisfaction and dropout rates in CETA 
classroom training programs illustrates these points. [29] On a general 
level, most persons were highly satisfied with participation in CETA. 
Further questioning revealed that the training they received was key in 
determining satisfaction. More detailed questions indicated that the 
following, specific aspects of the training program had an important 
influence on participant satisfaction (listed in order of importance): 

• Programs which seem like being on a job. 

• Instructors who emphasize student* s ability to do 
independent work. 

• Being evaluated positively by instructor. 

• Not having a problem with the instructor left unresolved. 

• Talking with the instructor frequently. 

• Frequent instructor feedback. 

• Having training plan discussed upon entry to school. 

The striking feature of this list is the important role that instructors 
play in shaping participant satisfaction. These findings underscore the 
importance of personnel decisions--obtaining and retaining competent, 
dedicated staff necessary for providing quality services to 
participants—in running successful programs. 

Does this mean that only dissatisfied participants drop out of the 
program? Findings from the study reveal that CETA participants who drop 
out are considerably less satisfied with their classroom training 
programs than those who complete them. Yet, among those who were 
extremely dissatisfied with their program, more than half "stuck it out" 
and completed the program. Thus, low satisfaction with training does not 



always result in dropping out. In fact, only a moderate number of 
participants dropped out because of dissatisfaction. The table below 
shows the main reasons for dropping out of training: 
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MAIN REASONS GIVEN FOR DROPPING OUT 
OF CLASSROOM TRAINING 



REASON 



PERCENT 



Financial Problem 
Received Job Offer 
Changed Vocational Goals 
Expelled from School 
Bored With Training 
Family Change 
Health Problems 
Behind in School Work 
Legal Problems 



24.6 
24.0 
10.3 
9.7 
8.0 
8.0 
8.0 
6.9 
1.1 



The table indicates that personal factors—not the program— are the main 
reasons for dropping out. Other data show that a substantial number of 
dropouts leave the program quite satisfied. 

In summary, these findings indicate satisfaction is a useful way of 
capturing participants' views about the quality of a program. However, 
satisfaction is not related to whether a participant completes the 
program. Knowing a program's dropout rate may say very little about how 
satisfied participants are about the program. 



GUIDELINES 

In this section detailed questions are presented to determine whether the 
participant is satisfied with specific services he or she received along 
the pathway, from entry to exit (or point of dropout). 

Questions are provided on the next pages for illustration. [14] Because 
services differ from one SDA to another, the questions should be modified 
to correspond with the services and programs along the pathway in your 
SDA. For example, the list of services in Question 2 may be expanded to 
match the menu of services offered to participants in your SDA (or 
contract organization), or questions for specific services (e.g., 
instructor evaluations for classroom training) may be added. 

These questions are intended to be asked when the applicant or 
participant drops out or terminates from the program. Depending on the 
reading and writing skills of participants in your SDA, the questions may 
be completed by the participant or asked by staff over the telephone or 
in an "exit interview." 



^i^? r H^"?^" these questions? If the questions were given to only a 
ov^iIi/?rt °'V\'^°'T^ ^"""P^" °^ "-^^ responses 'would indicate 

?^rcliif^''l''^'°: participants. However, because of 

the small number of clients completing the questionnaire, you could not 
determine for example, how satisfied clients in a specific target qrSSp 
were, or how satisfied clients at a given subcontractor were. Therefore 

' ideally, answer these questions. Then data would be 
l?rl l\ t T^"'" satisfaction levels across specific client groups? 
traJnfnn rf \^'°"P'- ^^Pf °^ draining received (OJT. institutional 
^?nn?I^: etc ) service provider (subcontractor), or for clients having 
single vs. multiple component training plans. 

Why should an SDA be interested in performing such analyses? Mainly 
because SDAs--even those with excellent overall performance records-- 
?nH- -H f P'-oblems placing specific kinds of participants. For these 
ndividuals. answers to the questions would provide useful feedback 
information for pinpointing problems in the pathway causing poor 
performance. Another reason is that SDAs and contractors are often 
interested in maintaining and improving the quality of their services. 
Answers to the questions may also be used to target staff efforts at 
improving those services with consistently low satisfaction levels across 
most client groups In short, through their mission and work components. 
SDAs and subcontractors devote much effort to planning effective 
employment and training services. Sometimes services are implemented as 
planned; sometimes not. Obtaining feedback on participant satisfaction 
IS a useful cross-check on the organization's implementation success. 

How should an SDA or subcontractor analyze these data? The previous 
ZTnVnT. kinds of responses, circled (multiple choice) 

^hf • , ^""^ ^^""^ ^^'^'^^ reasons for participant dissatisfaction). 
The circled responses may be added to the participant's data in the MIS 
statistical software may be used to produce tables, correlations, and 
other analyses for a variety of client groups as specified by the 
example, separate frequency distributions for the questions 
could be developed for participants in each training program. The text 
responses must be methodically reviewed and summarized by hand. The aim 
here is to determine the major reasons why participants are dissatisfied 
with a service, and to use this information to make service improvements, 
or to develop new, innovative ones. 

The satisfaction data can also be related to such factors as job 
retention and earnings, asking the question, "Do satisfied placements 



earn 



more and retain employment longer than dissatisfied ones?" Simple 
correlation or cross-tab tables may be used to perform this task. 
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• EXHIBIT 14 
SAMPLE PARTICIPANT SATISFACTION QUESTIONS 



Q-1 In general, how satisfied do you feel with the SDA? (Circle the number o1 
your answer. ) 

1 EXTREMELY DISSATISFIED 

2 SOMEWHAT DISSATISFIED 

3 NEITHER SATISFIED NOR DISSATISFIED 

4 SOMEWHAT SATISFIED 

5 EXTREMELY SATISFIED 

Q-2 Here is a list of services that you may have received at the SDA. Hov 
satisfied have you been with each service? 



CIRCLE YOUR ANSWER 



SOMEWHAT VERY 
VERY SOMEWHAT DIS- DIS- DID 

SERVICE SATISFIED SATISFIED SATISFIED SATISFIED NOT GET 



A. 


Application interview 


V-S 


s-s 


S-D 


V-D 


DNG 


B. 


Job market information you 
received 


V-S 


s-s 


S-D 


V-D 


DNG 


C. 


Developing your training 
plan 


V-S 


S-S 


S-D 


V-D 


DNG 


D. 


Training 


V-S 


S-S 


S-D 


V-D 


DNG 


E. 


Placement services 


V-S 


S-S 


S-D 


V-D 


DNG 


F. 


Support services 


V-S 


S-S 


S-D 


V-D 


DNG 



Were there any services that you wanted or expected but did. not get? 



1 f^O 

2 YES 



(If YES) What were they? 



About how many times have you met with staff in-person, face-to-face 
since you filTed-aut an application form? ' 

1 NONE 

2 QNGE 

3 TWO TQ FIVE TIMES 

4 SIX TO TEN TIMES 

5 OVER TEN TIMES 



How many times have you talked on the telephone with our staff since 
we first interviewed you? 

1 NONE 

2 ONCE 

3. TWO TO FIVE TIMES 

4 SIX TO TEN TIMES 

5 OVER TEN TIMES 

How satisfied were you with how fast you got service after ycu applied 
for them? 



1 VERY SATISFIED 

2 SOMEWHAT SATISFIED 

3 SOMEWHAT DISSATISFIED 

4 VERY DISSATISFIED 

How satisfied are you with our staff? 



1 VERY SATISFIED 

2 SOMEWHAT SATISFIED 

3 SOMEWHAT DISSATISFIED^ 

4 VERY DISSATIFIED — ^ 

(If dissatisfied) How can our staff improve? 



Q-8 How satisfied are you with the location of our office? 



1 VERY SATISFIED 

2 SOMEWHAT SATISFIED 

3 SOMEWHAT DISSATISFIED 

4 VERY DISSATISFIED 



(If DISSATISFIED) Please list the major reasons you are 
dissatisfied with the office's location. 



Q-9 How easy or difficult has it been to get in touch with our staff when 
you needed to talk with them or see them? 

1 NEVER TRIED TO CONTACT STAFF 

2 VERY EASY 

3 SOMEWHAT EASY 

4 SOMEWHAT DIFFICULT 

5 VERY DIFFICULT 

(If DIFFICULT) Please describe the difficulties you had 
trying to contact staff. 



Q-10 Are you employed? 

1 NO ^ GO TO BOTTOM OF PAGE 

2 YES 

Q-11 Did the services you receive help you find a job? 

1 VERY HELPFUL 

2 SOMEWHAT HELPFUL 

3 NOT HELPFUL AT ALL 

Q-12 How satisfied are you with your job? 

1 VERY SATISFIED 

2 SOMEWHAT SATISFIED 

3 SOMEWHAT DISSATISFIED 

4 VERY DISSATISFIED 



(If DISSATISFIED) Please list things about your job that 
you don't like. 



Thank you for completing our questionnaire! (Note: These questions may be 
asked at an exit interview, in a telephone interview, or a mail survey.) 
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THE JTPA CONNECTION 



The ultimate aim of JTPA services is to match employers with participants 
in the labor market. When examined separately as discussed earlier 
employers and participants may be satisfied with services leading up to 
the placement. But substantial dissatisfaction and low job retention 
rates may arise if participants are improperly matched with employers— a 
possible event, given the highly discretionary nature of the matching 
process. 

Satisfaction with the matching process is determined by categorizing 
placements as shown in Exhibit 15. For participants, Q-12 may be used as 
a general measure of overall satisfaction-dissatisfaction with the 
matching process. For employers, use Question 1. The JTPA connection is 
working at its best when most participants are placed, and when most 
placements fall into Cell A. 



EXHIBIT 15 



EMPLOYER-PARTICIPANT COMBINATIONS 



PARTICIPANT 



EMPLOYER 







NOT 




SATISFIED 


SATISFIED 


SATISFIED 


A 


B 


NOT 






SATISFIED 


C 


D 



If most placements fall into Cells B, C, or D, the matching process is 
not working well. Further analysis is needed to determine the reasons 
why. These analyses are often exploratory but should center on answering 
the following questions: 

• There are many types of placements, OJT, JSA, Job 
Development, etc. Is the pattern of employer/participant 
dissatisfaction the same for all types? 

• For placements with dissatisfied participants and/or 
employers, examine their responses to the other satisfaction 
questions. Are employers generally satisfied with 
participant traits? What are the major things that 
participants do not like about their job? 

Answers to these questions should identify those services/programs having 
the most empl oyer and parti ci pant di&sati sf acti on wi th the matchi ng 
process. Once targeted, the SDA's (or subcontractor's) program designers 
are in an informed position to make changes that may correct the 
situation. 



VALUES AND LEADERSHIP 



Before concluding this chapter, a final source of potential conflict in 
the SDA and subcontractor is considered— value conflicts among the SDA's 
leadership. JTPA is a partnership between the private sector and local 
government. We might expect private-sector dominated SDAs to favor 
serving the labor market, while SDAs dominated by local governments to 
favor the hard-to-serve. This has not been the case. [35] Both tend to 
have programs oriented to the needs of local labor markets. Yet, 
conflicts IP the partnership may arise over this and other issues. One 
focus of the social component, therefore, is determining whether the 
partnership is typified by consensus or controversy. Keep in mind, 
however, that conflict is not necessarily bad. It is natural for PICs 
and local governments to disagree at times, and conflict is often the 
only way of resolving these disagreements. The point here is to under- 
stand these conf 1 i cts as they affect SDA achi evement of goal s and 
performance standards. 

A second leadership issue is PIC turnover rates. Quality programs are 
likely to emerge when SDA leadership remains fairly stable across years. 
However, turnover and low attendance among PIC members is common. [35] 
These problems are mainly the result of excessive time commitments and 
state reporting and monitoring requirements. If this pattern continues 
to persist in SDAs, continuity in leadership (and, hence, the values and 
mission of the SDA) may deteriorate, and the quality of programs may 
decline. 



DISCUSSION 

In performing assessments of staff, employer, and participant, 
satisfaction, several factors should be kept in mind. First, SDAs 
subcontractors may operate pathways designed to create a certain amount 
of dissatisfaction among participants. This is because SDAs typically 
have insufficient resources to serve all eligible applicants. Intake is 
characterized by waiting lists, attendance requirements, homework 
assignments and other sorting mechanisms to separate the less motivated 
from the highly motivated, the job-ready from the unskilled, and so 
forth. These sorting mechanisms can create certain amounts of dis- 
satisfaction among enrol lees. Therefore, the proportion of highly 
satisfied participants in an SDA may be relatively small. 

Second, SDAs operate in a variety of economic environments. Some are 
typified by growth and low unemployment, while others are just the 
opposite. In the latter case, participants may have a difficult time 
finding employment and, consequently, have relatively low levels of 
satisfaction with services. Although staff work to improve the quality 
of services, their efforts may have little effect on participant 
satisfaction because of local economic constraints. In short, economic, 
political, and social conditions in the environment may impose real 
1 imits on staff abi 1 ities to improve either participant or empl oyer 
satisfaction. 

Third, depending on their goals, SDAs may be able to maximize either 
participant or employer satisfaction, but maximizing both might be 
difficult. In most SDAs, services are oriented around local labor market 
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and employer needs. PICs (or other employer advisory group) define 
participant skills needed for successful performance in the job market. 
Participant services teach skills that meet the requirements of 
cenjpliDyers. In other words., employers, or the gatekeepers who control 
.entry into the labor market., set the standards for employment, and 
a)articipants are expected to conform with these standards (if they ever 
hope "to find a job). The focus is on the participant's "short comings," 
and the need to "repac-kage" him or herself to i^e more "competitive." 
^Jnder 1;hi« mission, we might expect employers to have high levels of 
satisfaction as services are oriented around their interests. 
•Participants, who must conform with these interests, might have 
relatively lower levels of satisfaction. 

In terms of our SDA-suhcontractor problem, we find that shortly after the 
SDA increased its compliance efforts., so did the level of dissatisfaction 
among subcontractor staff. This increase was suffijcient to cause a 
25 percent turnover iti staff counselors. The subcontractor's performance 
jdaclined swith the introduction of replacement staff, who — just like their 
pradecessors^ — ^^were soon experiencing dissatisfaction with their new jobs. 
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CHAPTER 7 
HOW TO INTERPRET YOUR FINDINGS 

Overview of the Chapter 
Step 1: Collect Findings from Previous Chapters 
Step 2: Determine the Causes of the Problem 
Step 3: Design a Solution to the Problem 
Step 4: Distribute Your Findings 
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CHAPTER 7. HOW TO INTERPRET YOUR FINDINGS 



Let us begin by briefly reviewing where we are in the process evaluation. 
In Chapter 1 an SDA-subcontractor problem was described and a process 
evaluation was launched to discover the causes of the problem and find a 
solution. An organizational systems model, described in Chapters 1 
and Zy was used to conduct the process evaluation. In Chapters 3 
through 6 guidelines were presented for examining the mission, work, 
coordination and social components of the model * s conversion process. 
These guidelines were applied to the problem introduced in Chapter 1, and 
each component had its own set of findings. 

In this chapter, the major findings from all four components are 
summarized and inserted into an exhibit. (See Exhibit 16.) By comparing 
the findings across the four components, we will be able to define the 
causes of the SDA-subcontractor problem and propose a workable 
solution(5). Guidelines are presented illustrating how to perform these 
comparisons. Thus, the focus of this chapter is DESIGN— changing the 
parts of the organization to achieve a proper fit and thereby increase 
its effectiveness and efficiency. 



STEP 1: COLLECT FINDINGS 

The first task is simply to collect the major findings from Chapters 3 
through 6 and insert thera into a table. Because our examinations of each 
component probably uncovered truckloads of information, it is critical 
that only the major conclusions be inserted into the table. This is done 
not only because of limits on the table's size, but also because it will 
help us compare and interrelate the findings across components in later 
steps. Exhibit 16, which summarizes the previous findings for our 
SDA-subcontractor problem, is presented on the next page. 
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EXHIBIT 16 



SUMMARY OF MAJOR FINDINGS 



COMPONENT 



WITHIN 
CONDITION 



BETWEEN 
CONDITION 



MISSION Satisfied in part but revenue, 

al locative and treatment 
decisions, and performance 
standards are inconsistent 
with goals. Performance 
standards dominate goals. 



Satisfied in part; goals and 
access decisions are con- 
sistent with local conditions. 
Performance standards and 
revenue , al 1 ocati ve and 
treatment decisions are con- 
si stent wi th each othar but 
are inconsistent with local 
conditions. 



WORK Satisfied; steps in the SDA's 

competency-based system are 
carefully interrelated with 
each other. 



Satisfied; work component is 
consistent with mission 
component's goal to provide 
conprehensive services. 
However, dominance of 
performance standards i n 
mission component forces 
SDA to rely on intake 
screening mechanisms to 
sort out less motivated 
appl i cants. Service 
creaming occurs periodically. 



COORDINATION Satisfied in part. Although 
the mission and work 
components have mechanisms 
to coordinate their re- 
spective activities, the 
JTPA connection is not 
working properly. 



The first condition is 
satisfied; mechanisms exist 
to coordinate activity in the 
work component with activity 
in the mission component- 
The second condition is noL 
satisfied; few mechanisms 
exi St to coordi nate the 
SDA's activity with the 
subcontractor^ s activity 
in three areas: (1) pigeon- 
holing process; (2) SDA- 
subcontractor referral s ; 

and (3) participant progress 
through ITPs. 



SOCIAL Social consequences of above findings are substantial. High 

levels of dissatisfaction exist among subcontractor staff with 
25 percent turn-over rates. 



EKLC 
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STEP 2: DETERMINE THE CAUSES OF THE PROBLEM 



The next task is to examine the findings in Exhibit 16 to determine the 
causes of the SDA-subcontractor problem. This is performed using the two 
conditions that organizations must satisfy to operate well (as described 
in Chapter 2): 

(1) All four components must be involved in the conversion 
process. 

(2) A proper fit must exist among the four components. Proper 
f i t i s achi eved when the f ol 1 owi ng condi ti ons are met : 

a. The environment and the SDA's mission component must be 
consistent with each other. 

b. The four components of the conversion process must be 
consistent with each other. 



FIRST CONDITION: INVOLVEMENT IN THE CONVERSION PROCESS 

Let us begin with first condition above. Here, our task is to review 
each component, one-by-one, and determine whether it is involved in the 
conversion process. 

MISSION COMPONENT . The review begins with the mission component. We 
find that the mission component is clearly involved in the conversion 
process. That is, the SDA has well-defined goals and access, treatment 
and allocation decisions, as well as a fully-developed performance 
control system. These state the purpose , or mi ssi on , of the 
organization, giving it direction. 

On the other hand, if an organization's goals were either missing or 
poorly defined, it would be adrift without a clear sense of purpose and 
direction. In this case, we would conclude that the mission com^jonent 
was jTot involved in conversion process. This would be a major problem in 
most organizations (with stable environments). 

Corrective actions would consist of an intensive planning effort to 
establish the component's elements in the conversion process. 

WORK COMPONE NT. For the most part, the work component is also an 
integral part of the conversion process. The exception occurs in service 
creaming, or when a participant receives a less costly service than the 
one prescribed in the ITP. However, as the work component's BETWEEN 
condi ti on i ndi cates (i n Exhi bi t 16) , thi s f aul t i s caused mai nly by 
performance standards, which dominate the mission component. 

How can you tell when the work component is not involved in the 
conversion process? Normally, this generally occurs when few applicants 
are enrolled, or when applicants are enrolled but actually receive few 
(if any) services. In essence, it indicates that the organization lacks 
means of accomplishing its goals. 
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COORDINATION COMPONENT . This component is not fully involved, as 
indicated by (1) the lack of SDA-subcontractor coordination mechanisms in 
several areas; and (2) problems in matching employers and participants. 

SOCIAL COMPONENT . Here, the question is not so much whether the social 
component Ts ""involved" in the organization, but whether the other 
components have caused tension and conflicts that threaten the 
organization's survival. These issues will be examined later in the 
second condition (Part B). 

CONCLUSIONS . Our review of Exhibit 15 findings reveals that the first 
condition is not satisfied for the coordination component, and that the 
coordination component is a major cause of the SDA-subcontractor problem. 

SECOND CONDITION: PROPER FIT 

PART A . To determine whether the environment and the SDA's mission are 
consistent, we must examine the first row of Exhibit 15. On the one 
hand, the SDA's goals and access decisions are consistent with each other 
and local conditions. On the other hand, performance standards dominate 
over the SDA's goals and access decisions, with revenue, treatment, and 
allocative decisions being consistent with the standards. Thus, the 
mission component is inconsistent with its environment, and this 
condition is not satisfied. The implications nere are that while the SDA 
will likely achieve its performance standards, it will probably not 
achieve its goals. 

This inconsistency contributes to the SDA-subcontractor problem in an 
i ndi rect way. Because performance standards domi nate the mi ss i on 
component, the SDA places heavy emphasis on its performance control 
sys^em and compliance efforts. This, in turn, only aggravates the co- 
ordination problem. 

PART B . Organizations satisfy this condition when their work, 
coordination, and social components are consistent with each other and 
with the mission component. This is done by comparing each component 
with all the others. For the most part, this condition is not satisfied 
for the following reasons. 

First, and most importantly, the mission component calls for SDA 
provision of intake services and subcontractor provision of training and 
employment services. No mechanisms exist in the coordination component 
to carry out this charge. Further, no workable mechanisms exist in the 
coordination component to achieve the SDA's second goal of client- 
employer matching. In short, the mission and coordination components are 
inconsistent. And because the mission component largely determines the 
shape of service delivery in the work component, the coordination and 
work components are also inconsistent with each other. 

Second, as explained in Chapter 5, the work component is consistent with 
the mission component but in a curious way. The work component's 
competency-based system is consistent with the SDA's goal of compre- 
hensive services. However, the dominance of performance standards in the 
mission component push the SDA and subcontractor toward short-term, 
single service programs and service creaming. In other words, the work 
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component is being pulled in opposite directions by the goal-performance 
standard inconsistency. The major consequences here are (1) increased 
conflict, stress and staff turnover in the social component, and, as 
mentioned earlier, (2) SDA goals may not be achieved. 

In short, a proper fit has not been achieved among the mission, work and 
coordination components. This, in turn, has caused severe levels of 
tension and dissatisfaction among subcontractor staff, has produced 
SDA-subcontractor conflicts, and has threatened the very survival of the 
subcontractor. Specifically, the following adverse social consequences 
are occurring: 

• MISSION COMPONENT: The dominance of performance standards 
results in increased SDA compliance, which subcontractor 
staff view as unjustified harassment (that does not solve 
their problems). 

• WORK COMPONENT: Staff stress is caused as the organization 
is pulled in opposite directions. On the one hand, staff 
prefer helping participants overcome their individual 
employment problems through comprehensive service plans, but 
the dominance of performance standards pushes staff to 
assign short-term, one-shot services. 

• COORDINATION COMPONENT: No matter how hard SDA and 
subcontractor staff work, the subcontractor's problems do 
not improve because few coordination links exist between the 
agencies. This causes staff frustration and 
dissatisfaction. 

• SOCIAL COMPONENT: As a consequence of all of the above, the 
subcontractor experiences high levels of staff turnover. 
Newly hired staff soon become dissatisfied with their work. 



CONCLUSJONS 

Pulling together the evidence presented in this step, we find the 
following causes of the SDA-subcontractor problem: 

1. The coordination component is not involved fully in the conversion 
process. 

2. The coordination component is inconsistent with the mission 
component. 

3. The elements of the mission component (mainly goals and performance 
standards) are inconsistent with each other. 

4. As a consequence (of Number 3) the work component is being pulled in 
opposite directions. 

5. All of the above have produced a severe level of stress, tension, 
and conflict in the social component. 
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STEP 3: DESIGN A SOLUTION TO THE PROBLEM 



The next task is to design a solution to the organization's problems. 19/ 
This is accomplished by changing the mission, work and coordination 
components so that the two conditions described in Step 2 are satisfied. 
For our SDA-subcontractor problem, these changes in the conversion 
process must address and nullify the five causes listed above. The 
following changes are recommended: 



• COORDINATION COMPONENT. The most critical solution is to 
involve the coordination component in the conversion 
process. That is, now that the SDA has well-defined mission 
and work components, it should shift its focus to inter- 
agency coordination issues. Following extensive meetings 
between the SDA and subcontractor, the following actions 
were taken to improve SDA-subcontractor coordination. These 
resolve the first and second causes of the problem and 
partly eliminate the fifth cause. 

1. As a requirement for all contracts, the subcontractor 
must provide services consistent with the SDA* s 
competency-based system (i.e., achieving coordination 
through standardization of skills across agencies). 
Thi s greatly reduces subcontractor compl ai nts about 
referred participants. 

2. Given a knowledge of the competency-based system, the 
subcontractor is responsible for monitoring the 
participant's progress through training, and providing 
this information to SDA intake staff. 

3. SDA intake staff become responsible for placement and 
follow-up services. This allows one set of staff to 
know both participant and employer needs, resulting in 
better, more compatible participant-employer matches. 

4. Each participant's sequence of services in the ITP is 
newly stored in the MIS. When it is time for a 
participant to start a new service in the sequence, the 
MIS prompts both the appropriate intake and sub- 
contractor staff members that a referral should occur 
shortly. This reduces the number of participants who 
fall through the wide-open cracks in the system. The 
ul timate effect of these changes i s i ncreased 
efficiency and improved SDA-staff relations. 



19/ It is beyond the scope of this guide to describe all possible 
solutions to all possible problems that an SDA or subcontractor may 
experience. Hopefully, the guidelines presented here are sufficient 
for handling your specific case. 
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• MISSION AND WORK COMPONENTS. The third and fourth causes of 
the problem concern the mission component and its effects on 
the work component. If the inconsistencies in the mission 
component can be eliminated or reduced, problems in the work 
component may also disappear. The key issue here is the 
lack of consistency between the SDA's goals and performance 
standards. There are several ways of making the SDA's goals 
and performance standards consistent with each other, such 
as: 

1. Change the SDA's goals so they reflect the performance 
standards. In our case, however, this should not be 
done because the goals would then be inconsistent with 
local conditions. 

2. Reduce the SDA's performance standards so they reflect 
the goals. This makes sense, because the SDA could 
allocate more resources toward fewer participants, 
making comprehensive services possible. 

3. Increase the SDA's revenues. If the SDA had more 
resources, it could provide comprehensive services to 
clients and still meet performance standards. 

In our case, both Number 2 and Number 3 were implemented. The state, 
after reviewing the results from the SDA's process evaluation, lowered 
its performance standards and also provided additional six percent funds 
to handle the situation. In addition, the SDA boosted the sub- 
contractor's revenues through a re-allocation of its own funds. 

Finally, as a consequence of these actions, SDA-subcontractor relations 
improved greatly, and stress, tension and conflicts were reduced. 



STEP 4: DISTRIBUTE YOUR FINDINGS 

Many SDAs may be reluctant to share the results of their process 
evaluations. There may be fears that information collected during the 
evaluation might hurt the agency in some way in the future. While these 
feel i ngs can be understood, there are good reasons for shari ng your 
results with others throughout all levels of JTPA. The most important 
reason is that, in many ways, SDAs are islands. A manager of an east- 
coast SDA, for example, might have solved a problem that is just emerging 
in a west-coast SDA. If the west-coast SDA knew about the other SDA's 
experience, the amount of effort required to solve the problem could be 
reduced greatly. And the aggravation of "continually re-inventing the 
wheel" would disappear. Therefore, SDAs are encouraged to share their 
findings with other SDAs in their respective states as well as across the 
country. 

Congress should also be interested in these findings for two reasons. 
First, SDAs with very different goals, performance standards, and 
participants can be compared to determine whether "returns on investment" 
are similar or vary across SDAs. Each SDA's return on investment can be 
determined by calculating cost-benefit and cost-effectiveness ratios. 20/ 
One SDA, for example, might spend a little money on a lot of people, and 
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its net gains per placement may be small. Another SDA may train less 
people more intensively; the returns to each person may be large. In 
each case the cost-benefit ratio could be the same and each program could 
represent reasonable approaches. However, if creaming is rampant, some 
of the returns of the lovt-cost, quick placement strategy are due simply 
to the characteristics of the participants, not the . services they 
received. 

Second, with over 500 SDAs nationwide, it is unlikely that JTPA will 
succeed in all places. By collecting SDA process evaluations from 
several sites. Congress could pinpoint and understand the conditions 
associated with program success and failure. Specifically, as described 
in Chapter 1, programs generally fail because of two main reasons: 
(1) the implementation strategy may not work in the field, or (2) the 
program* s services may not work as intended. In reviewing the 
achievements of JTPA, Congress could draw from SDA process evaluations to 
determine how these two factors influenced program outcomes. The studies 
would also help Congress evaluate whether its own, underlying assumptions 
were, in fact,^ correct. In the end, the studies would probably indicate 
that substantial program variation exists across SDAs, making simple 
blanket endorsements ("it works") and wholesale rejections ("it doesn*t 
work") less appropriate and meaningful. At best, such results could 
assist Congress in improving the design of employment and training 
programs in the future. 



20/ These techniques are described in Volume VII Issues iji Evaluating 
Costs and Benefits. 
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APPENDIX A 
JTPA IMPLEMENTATION STRATEGY 



To understand JTPA's implementation strategy, we need to understand why 
Congress designed JTPA the way it did, and how Congress expected JTPA to 
wov;k. That is, why did Congress believe that a public-private partner- 
ship was important? Why were performance standards established? Why the 
emphasis on training and related services? Why the allocation of 
40 percent of the funds for youth and 70 percent of adult funds for 
training? And, why the state coordination and labor market information 
role in the program? 

A review of JTPA's legislative history indicates that, as CETA neared 
expiration, the House, Senate and the President proposed separate 
employment and training bills to replace the CETA program. JTPA emerged 
as compromise legislation from the competing interests of these and other 
groups. In many ways, then, JTPA represents the collective wisdom of 
lawmakers at this point in history. Whether Congress and the President 
were correct remains to be seen. But answers to these and other 
questions may provide important clues regarding Congressional intent and 
also serve as benchmarks for interpreting findings in SDA process 
evaluations. 

Why a partnership? Congress felt that under CETA, training was not 
linked sufficiently to real, long-term jobs in the private sector. 
Training occurred for jobs that did not exist. This perceived deficiency 
was due, in large part, to insufficient private sector participation in 
the development of job training programs. Bi-partisan support existed 
for an enlarged Private Industry Council (PIC) role under JTPA compared 
to CETA to attract business leaders to the program. Ideally, PIC 
busi ness members woul d have more accurate i nsi ghts i nto trai ni ng and 
labor force needs in a local labor market—would have a finger on the 
pulse of the business community— in ways that Prime Sponsors alone could 
not. It was^ also hoped that PIC business members would offer training 
slots in their own businesses and, through their influence, help develop 
training slots in other companies. The chief role of the public sector 
in the partnership was to assure accountability for expenditures of 
public funds. [1,3] In short, private sector participation on PICs would 
make the JTPA connection work. 

Why Performance Standards ? The chief reason for performance standards 
was to assure accountability and efficient use of funds. [1-5] Congress 
intended that the performance standards evaluate not only achievement of 
outcomes but also relative gains made by participants in reaching 
outcomes. Real concern existed that the performance standards might 
pressure service providers to "cream," for it was "...not the intention 
of this act to encourage (SDAs) to serve only the most job-ready." 
[4, page 11] Some performance standards, such as cost-per-participant , 
were criticized because it might discourage long-term training, which 
could have significant impacts on outcomes. In short, while performance 
standards were essential for accountability, they might also have a great 
influence on who receives what services in the program. Hence, Congress 
expected the Secretary to moni tor the appl i cati on of performance 
standards and "fine tune" them (where necessary) in a manner consistent 
with Congressional intent. 
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Why the 70 percent/30 percent allocation of Title II-A adult funds ? The 
70 percent allocation to training services was the mechanism Congress 
developed to keep all non-training activities to a minimum. With 
15 percent of the remaining funds for administration (the 20 percent 
allotment under CETA was considered too generous), 15 percent remained 
for support services to assure that persons with no income could still 
participate in the program. Concern was expressed, however, that the low 
stipend percentage might push disadvantaged into short-term training, 
leading to low-paid, dead-end jobs. [1, 4, 10] 

Why the 40 percent youth allocation ? Congress i ntended youth to be 
funded at levels similar to CETA. Congress perceived that early inter- 
vention for disadvantaged youth is socially more cost-effective, based on 
evidence that unemployment at an early age is associated with long-term 
unemployment as an adult. [1, 2, 5] 

Why the state role regarding coordination and labor market information ? 
Allocation of funds and coordination by states was perceived to be the 
most efficient organizational design for job training programs. State 
coordination of training services would help avoid duplication of 
services and establish linkages with other programs. State allocation of 
funds, such as incentive and set-aside funds, would encourage SDAs to 
develop special services or service groups that otherwise might be 
ignored (e.g. , to encourage delivery of basic education ski 1 Is and 
remedial education services to prepare disadvantaged or a target group 
for later occupation-specific training). States would also provide more 
efficient and coordinated production of labor market information to 
improve matching of training to labor market slots. [1, 4] 

In summary. Congress designed JTPA to correct problems and abuses under 
CETA as well as to establish new employment and training policies for the 
country. Striking differences exist between the implementation of the 
two programs. Yet, Congress made only one major change in the types of 
services offered to JTPA participants--the elimination of Public Service 
Employment (PSE). This indicates that if JTPA is an improvement over 
CETA , the improvements wi 1 1 happen mai nly through changes i n impl e- 
mentation, a.id how JTPA's implementation influences service delivery. 
This emphasis on implementation underscores the importance of state and 
SDA process evaluations in understanding program performance. But, how 
is JTPA performing? Are states and SDA's honoring Congressional intent? 
This is the subject of the next section. 



JTPA REALITIES 

Recent JTPA implementation studies suggest that, in general, JTPA service 
del ivery systems across states are working productively. [15-18] 
However, the Congressional design and funding of JTPA have resulted in a 
number of equity issues— that is , questions about who receives JTPA 
services. Under JTPA, business-dominated state councils and SDA PICs 
were created to establ ish service pol icies in their respective 
jurisdictions. A major policy—or access decision--confronting the 
councils and PICs was, ''Whom should the program serve?'' The two basic 
choices were: 
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1. Serving those with the greatest training needs. 



VS. 

2. Serving the labor market needs of business (i.e., 
assuring that training programs are responsive to the 
needs of employers for a well-trained labor force). 

The majority of councils and PICs have chosen the second policy. This 
choice, however, was also partially the result of the following program 
characteristics in the law: 

• Inadequate funding: on average, SDAs have sufficient fun^s 
to train only four percent of all eligibles in their 
respective areas. 

• Spending limits: Limitations on spending for work 
experience and support services, plus a lack of stipends. 
These services are commonly used for youth and adults with 
the greatest needs. Service providers are also unable to 
retain participants due to the limits. 

• Spending requirements for youth (although these are often 
waived or reduced by states). 

• Performance standards: states and SDAs are required to meet 
standards , creati ng i ncenti ves to reduce ri s k by servi ng 
eligibles most qualified to meet employer needs. 

In short, serving employer and labor market needs and meeting performance 
standards, all within a limited amount of funding, encourage SDAs to 
serve eligibles who are easier to place. 

The following state council and PIC policies and implemeiitation 
procedures have also reinforced this policy: 

• Concern that a decline in JTPA performance levels (caused by 
training the hard-to-serve) would make the program seem less 
important, and perhaps not worth keeping, if further cuts in 
federal spending were contemplated. 

• A high priority of many councils and PICs is to improve 
public relations in the business community following CETA's 
termination. The best way to accomplish this aim is to 
serve employer needs. 

• Employers participating in JTPA training programs make the 
final decision whether a participant (who has completed 
training) becomes their employee. Thus , the private 
sector's access decisions greatly influence the flow of 
participants at both the front (input) and back (output) 
ends of the system. 

• In choosing between quantity (placing as many people in jobs 
as possible with a given budget) .versus quality (providing 
the best quality services to as many participants as funds 
permit), most PICs tend to favor quantity. 



Youth competencies are difficult to assess. Therefore, 
youth programs are often oriented for those 19-21 years 
old— those who are out of high school and old enough to work 
full-time, where the only outcome is a placement. Youth 
aged 16-18 receive relatively less attention in the program. 

No states (in the studies) imposed additional eligibility 
requirements on SDAs, but specific target group requirments 
were sometimes imposed. However, the more target groups 
that were identified, the less impact targeting would have, 
because fewer resources would be available to serve each 
target group. 

In most performance-based subcontracts, the SDA's 
performance standards and financial liability are passed to 
the subcontractor. Thus, subcontractors often share PIC 
incentives to serve the needs of employers. 



These same forces have also produced more similarity than diversity in 
local program priorities, services, participants and performance across 
local areas. [18] Faced with performance standards and low funding, most 
SDAs^ have designed their training programs and services around two 
critical factors, achieving high placement rates at a given 
target— usually a low cost. As a result, basic activities like classroom 
and on-the-job training are found everywhere. Few SDAs devote much money 
to support services or innovative and exemplary programs, or to designing 
programs targeted to groups with special problems. All of these have 
higher costs and greater risk of not meeting performance standards and, 
hence, are relatively avoided. 

These patterns indicate that JTPA is not designed and funded to serve 
those with the greatest training needs. In reality, most SDAs orient 
their programs around local employer needs. All of this results in 
fairly standard programs operating within the law. 
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PREFACE 



One fall, a small cricket found himself becoming colder and 
colder with each passing evening as the weather turned 
wintery. So he went to the wise old owl of the forest and 
said to him, "Oh wise old owl, please tell me what to do. 
The weather is getting colder and every evening I shiver and 
shake with the cold. If I don't do something soon, I know I 
will soon freeze to death. What can I do?" 

"The answer to that is simple," said the wise old owl. 
"Just turn yourself into a grasshopper, and hibernate for 
the winter." 

"But how can I turn myself into a grasshopper, oh wise old 
owl?" asked the cricket. 

Replied the owl, "Humph, don't bother me with the details. 
I've given you the principle. You implement it." [10] 

With passage of the lob Training Partnership Act (JTPA) in 1982, Congress 
created new principles for organizing and operating the nation's largest 
employment and training program. States would assume greater admini- 
strative roles. Services to economically disadvantaged would be orovided 
through local "service delivery areas" (SDAs). New partnerships would be 
formed between the private sector and state and local governments. 
Performance standards would be enforced. With the new principles, 
however^ came very few details on implementing JTPA. No "how-to" books 
or other resources existed to guide implementation. States and SDAs soon 
realized that they were "on their own," and that successful implement- 
ation of JTPA would be a learning process as they ventured into new 
administrative territory. 

This is, in essence, what evaluation is, a learning process , that can be 
used to improve JTPA performance. Formally defined, evaluation is the 
collection and analysis of information by various methods to determine 
the effectiveness and efficiency of JTPA activities. In its simplest 
form, evaluation involves carefully examining JTPA to identify program 
elements that do and do not work, at what cost and for what reasons. In 
short, evaluation helps decision-makers pinpoint and understand the 
conditions associated with program success and failure. 

When these conditions become known, identifying--indeed, creating--the 
conditions necessary for effective and efficient service delivery become 
possible. If state and local evaluations of JTPA were conducted nation- 
wide, they would likely indicate that substantial program variation 
exists across service delivery systems, making simple blanket endorse- 
ments ("it works") and wholesale rejections ("it doesn't work") less 
appropriate and meaningful. [15] 

However, many states and SDAs are unprepared to conduct evaluations of 
JTPA programs. Prior to JTPA, few states and local areas had performed 
evaluations of their employment and training programs. If each agency 
independently developed its own evaluation design, much duplication of 
effort and inconsistency in the designs might result. To correct this 



situation, the National Commission for Employment Policy funded the 
Washington State Employment Security Department to develop evaluation 
designs tor use at the state and local level. The designs are intended 
to provide guidance and some uniformity to JTPA evaluation efforts across 
states and local areas. 

OUTCOME EVALUATIONS 

In this project and most pro ^ ^m evaluations, two types of evaluations 
may be conducted, outcome and process. As its name implies, outcome 
evaluations determine the results (or "outcomes") produced by the 
program. Three major types of outcomes are often considered: proximate, 
final and distributive. Proximate outcomes refer to the immediate 
results of program efforts. For example, proximate outcomes for a 
welding training program might include getting a job after training, or 
getting a job in welding or in a related specialty. Final outcomes refer 
to the long-term results of the program— for example, whether an 
individual is still employed with increased earnings one year after 
training. Positive proximate outcomes do not show conclusively that a 
participant has improved his or her long-run earning capabil ity. 
However, their presence does suggest that individuals are progressing 
toward stable employment, while their absence may indicate little 
progress toward longer-run benefits. 

The last type is distributive outcomes, which define proximate and final 
outcomes for specific target groups (e.g., welfare recipients, 
minorities, etcT) or geographic areas (e.g., rural vs. urban). 
Distributive outcomes reflect equity issues in service delivery and are a 
source of variation useful in explaining program outcomes. 

Outcomes may be measured at the completion of training (proximate), one 
year after training (final) or for distinct groups of participants, such 
as the handicapped. By themselves, these measures are commonly known as 
"gross outcomes," or simple percentages, totals or averages describing 
JTPA performance. Gross outcomes may be the result of a variety of 
factors, such as the services participants receive in JTPA, the education 
or job history of participants prior to entering JTPA, or changes in a 
local economy. The central aim of an outcome evaluation is to differ- 
entiate among these alternative causes to isolate and measure those 
participant outcomes produced by JTPA alone. The latter measures are 
known as "net outcomes," and evaluations estimating these service effect$ 
on outcomes are known as "net impact evaluations." 

Thus, while gross outcomes merely document program results produced by 
any number and variety of factors, net outcomes pinpoint results caused 
by the program. Therefore, net outcomes are a more accurate measure of 
program performance than gross outcomes. 

PROCESS EVALUATIONS 

Outcome evaluations provide only part of the information needed in 
program evaluation. By their methods, outcome evaluations often treat 
programs as "black boxes": what goes on inside the program (or box) to 
produce the results is rarely assessed. Thus, while outcome evaluations 
inform us about the results produced by the program, they rarely explain 



why the results were found. Process evaluations (also known as imple- 
mentation assessments, or evaluations of program implementation) fill 
this knowledge gap by analyzing the processes that produced program 
results. The common aim of most process evaluations is to describe the 
black box— to specify the interventions and implementation strategy that 
caused the evaluation's outcomes. 

Two process evaluation guides— one for SDAs (and subcontractors) and one 
for states— have been developed in this project. The SDA guide provides 
an understanding of how SDAs are organized, or structured, how this 
structure shapes operations and outcomes, and how all of this is in- 
fluenced by the statewide JTPA service delivery system. Guidelines for 
analyzing organizational operations in SDAs and subcontractors are also 
presented. Although the guide is written mainly for SDA and sub- 
contractor administrators, state officials mav also find it useful for 
understanding "what goes on" at the SDA- level. ' 

In this volume the state process evaluation is described. Guidelines are 
presented for tracing the influence of state policies on SDA and sub- 
contractor operations. A key point in this discussion is that the 
characteristics of the statewide JTPA service delivery system severely 
limit, st ates abilities to perform process evaluations of their own 
policies. Unless the proper organizational supports are present , process 
evajuatjons of how state policies influence local operations become 
virtualj^ impossible to perform . The guideTmii, along with the 
requisite organizational supports, are presented in the remaining pages. 
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APPROACH 



BACKGROUND FOR THIS GUIDE 

In this section *'State Process Evaluation" is defined. In the next 
section, constraints that limit states from performing process 
evaluations are discussed. The final section presents methods for 
performing a state process evaluation within the constraints imposed by 
the JTPA service delivery system. 

In the process evaluation approach presented here, the focus is on state 
p^Vicies and how these policies influence JTPA activities at the local 
^ level. Ide^illy, policies established at the state level move down to the 
local level ,^ promoting more efficient, more effective, or more equitable 
service delivery. Sometimes, however, gaps emeroe beLwe^n state policy 
intent and local program realities. This g 'ide is designed to assist 
JTPA practitioners at all leve'is in bridging the gap. The process 
evaluation includes a set of gui^lelines for sysl. ^tical ly examining the 
consequences of state JTPA policies for local service delivery (i.e., 
SDAs and their respective subcontractors). [4] These guidelines may be 
anplied to policy issues 1/ in the major areas of state responsibility 
deiined in the Act. Exhibit 1 lists these responsiuilities and describes 
representative policy issues in each. 

Policy issues are not settled in a vacuum; they are developed and 
resolved in the statewide JTPA service delivery system. Appendix A 
contains a detailed description of this system. Some of its features are 
summarized below: 

• Multiple Organizations: In general, the JTPA service delivery 

system consists of the fol lowing loosely 
connected organizations: the state, 
service delivery areas, and subcontractors. 
Administrative and/or servicp coordination 
may occur among subcontractors , between 
subcontractors and SDAs, and between SDAs 
and the state. Other agencies, such as 
welfare, also become part of the system 
through formal coordination agreements; 

Each organization has its own distinct 
rights and responsibilities; and, 

Each SDA is free to develop its own 
programs (within the constraints imposed by 
the JTPA). 



Decentralization: 
Local Autonomy: 



1/ A policy issue is a disagreement or conflict about an actual or 
potential course of state action. [4] 
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EXHIBIT 1 



STATE RESPONSIBILITIES AND REPRESENTATIVE 
POLICY ISSUES 



1. Establish and operate the State Job Training Coordinating 
Council (SJTCC). 

• What functions should the SJTCC perform? 

• How does the mix of publ i c/pri vate membershi p i nf 1 uence 
SJTCC functions and decision-making? 

II. Designate service delivery areas. 

• Under what circumstances should SDA boundaries be changed? 
III. Coordinate JTPA services with other programs. 

• What is the best way(s) to connect JTPA to the mainstream 
of state and local services? 

IV. Award six percent (incentive) money. 

• What criteria should be used to distribute this money to 
SDAs? 

• Under what conditions should technical assistance be 
provided to SDAs? What form of assistance should this be? 

• Should SDAs be required to distribute a portion of their 
incentive funds to subcontractors? 

V. Establish and administer vocational education programs 
(eight percent set-aside). 

• What state agency should operate this program? 

• How should this activity be coordinated with SDA programs? 

VI. Establish and administer an older worker program (three percent 
set-aside). 

• What state agency should operate this program? 

• How should this activity be coordinated with SDA programs? 
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Establish and administer the Title III Dislocated Worker 
program. 

• What state agency should operate this program? 

• How should this activity be coordinated with SDA programs? 

Establ i sh and admi ni ster state 1 abor market i nf ormati on 
programs. 

• What information do SDAs need to develop their job 
training plans? 

• How often do they need this information? 

• Do subcontractors require similar or different 
information? 

Monitor the performance of SDAs. 

• Under what conditions should the Governor impose a 
reorganization plan? 

• How do state moni tori ng/ compl i ance requi rements affect 
local operations? 

Prescribe service target groups. 

• What target groups should all SDAs be required to serve 
(if a^y)? 

• How does imposing target group requirements on SDAs 
influence service delivery? 
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These features have major advantages for local areas and, indirectly, for 
states as well. SDAs and subcontractors may customize their programs to 
meet the unique needs of participants and employers in their respective 
labor markets. In principle, the more services are designed to meet 
local participant and employer needs, the more effective services should 
be. And the more effective the service, the more likely that local goals 
and performance standards will be met. This structure also promises high 
performance levels on an overall, statewide basis. 

The system's major disadvantage is that it creates severe problems for 
designing state policies and for conducting process evaluations of how 
these policies affect local operations. Because responsibilities are 
decentralized, states (with SDAs) often lack the necessary local 
information to design policies that meet local interests and needs. The 
numerous organizations in the system and their differing relationships 
create uncertainty that state policies can be transmitted properly to 
SDAs, subcontractors and other coordinating agencies. Even if state 
pol icies are explained thoroughly to service del ivery areas, local 
autonomy may sway some SDAs, PICs and subcontractors not to follow them, 
or block state efforts to monitor the policies. In short, the loose, 
multi-organizational structure of JTPA' s statewide service delivery 
system presents potential obstacles to states' abilities to conduct 
analyses of their policies. 

In the following section, these constraints are examined in-depth to 
derive workable approaches for conducting process evaluations within 
them. 
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CONSTRAINTS 



This section is divided into three parts. First, the statewide JTPA 
service delivery system is reviewed briefly. System characteristics that 
limit states' abilities to design policies and trace their effects on SDA 
operations are reviewed. In the second part, a framework for examining 
operations in SDAs and subcontractors is described briefly and its impli- 
cations for performing state process evaluations is discussed. A final 
part contrasts the characteristics of the prior two parts with the re- 
quirements of "perfect implementation" of state policies. This becomes 
the basis for the state process guidelines, which are presented in the 
final part of this guide. 

Statewide JTPA Service Delivery System 

The structure of the statewide JTPA service delivery system is a major 
constraint in state process evaluations. The JTPA service delivery 
system consists of several, loosely connected state and local organiza- 
tions, each having specific roles and responsibilities in the system. 
(See Appendix A for a more detailed description of the system.) A chief 
state responsibility is monitoring the performance of local areas through 
its "performance control system." (Appendix A also contains a further 
elaboration on the state performance control system under JTPA.) Using 
the DOL regression formula or other mechanisms, states establish perfor- 
mance standards for each SDA. The state monitors each SDA's progress 
toward meeting its standards through a management information system 
(MIS). Thus, performance control systems are concerned with overall SDA 
results for given periods of time--not with specific actions or 
decisions. 

This means that states have little detailed information about what goes 
on at the local level. The MIS offers few insights regarding SDA 
operations and problems. Resource constraints and the geographic 
separation of state and local agencies restrict states' abilities to 
learn more about local level activities through other means. In short, 
i f a state were i nterested i n exami ni ng how a pol icy affected SDA 
operations (i.e., by performing a state process evaluation), it might not 
have the tools to do so. In the next section a framework for examining 
SDA operations is presented to gain more insights into this evaluation 
problem. 
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How SDAs Work 

A complete overview of how SDAs work is presented in a companion volume, 
A Guide for Process Evaluations , Local Level , Some of its more salient 
features are presented here. Readers interested in a more detailed 
discussion are encouraged to review the SDA Guide. 

To begin, the SDA is just one organization in a larger environment that 
includes the statewide JTPA service delivery system, other service 
agencies (such as welfare) as well as the local labor market and the 
political-social setting. As an organization, the SDA is composed of 
several parts that work together to produce outcomes,- participants hired 
by employers. Governance— that is, the PIC and SDA management-"Control 
how the SDA^s parts work together. Governance's aim is to convert inputs 
from the envi ronment (i . e. , resources--such as staff, pav*ticipants , 
suppl ies, employers, etc. —and information) into outputs (i.e., 
participants hired by employers) as effectively and efficiently as 
possible. This conversion process is carried out according to the 
following four components: 

• Mission: This includes the goals, objectives and performance 

standards of the SDA. The planning process drives 
activity in the mission component. Under govern- 
ance's direction, services are chosen, service 
providers are selected, and resources are allocated, 
taking into account local conditions and participant/ 
employer needs. Implementation of the plan is 
monitored through the SDA's performance control 
system. 

• Work: This includes the SDA's means of achieving its goals. 

It includes the training participants receive along 
the path they traverse through the system. Under 
subcontract arrangements, this component is performed 
by another organization. 

• Coordination: This includes the coordination links between the SDA 

and other agencies; the coordination of activity 
wi thi n the SDA ; communi cation patterns ; rul e-maki ng 
processes; and the process for matching participants 
with employers, 

• Social: This includes participant, employer and staff 

satisfaction and morale; the needs of participants, 
empl oyers and staff ; and the val ues of the SDA 
(particularly as reflected in the SDA's goals). 

Through the mission component's performance control system, 
governance--the PIC and SDA managements-receives feedback regarding how 
well the organization is performing. If the governance receives feedback 
i ndi eating a discrepancy between goals (or performance standards) and 
outputs, it may alter the components of the conversion process to 
minimize or eliminate the difference. 




In SDAs (and virtually all organizations), the key to good performance is 
achieving a proper fit among the parts (of the conversion process)— 
making sure they are internally consistent, meshing together in synchrony 
to produce outcomes consistent with SDA goals and local conditions. In 
states with two or more SDAs, local autonomy assures that each SDA is 
(relatively) free to design its own organization— that is, deciding how 
the parts of organization (described above) will fit together. There- 
fore, no two SDAs are exactly identical. 

This decentralization of JTPA has two important implications for process 
evaluations of state policies. First, as stated earlier, because state 
administration is usually geographically distant from SDAs (and is a 
separate organization in most states), states usually know little about 
each SDA's conversion process--that is, the contents of each component 
and whether a proper fit has been achieved, although long-term poor 
performance is a good indicator of failure to achieve such a fit. 

Second, because no two SDAs are identical in their organizational 
structures and processes, state policies will likely affect each SDA 
differently. Consequently, states— or anyone else for that matter— may 
be unable to predict (with certainty) whether a policy, when implemented, 
will work as intended. 

In terms of this model, three outcomes are possible when a state 
implements a policy: (1) the policy improves SDA performance by 
improving the fit among the organization's parts; (2) the policy has no 
effect on the organization's fit and, hence, performance; or (3) the 
policy destroys the fit and performance declines. The first is the 
target of success for all policy; the second is an undesirable but still 
acceptable outcome; the third is to be avoided at all costs. How hard is 
it to develop a policy that is guaranteed to have the first outcome? 
Very hard, and virtually impossible in JTPA and most public programs. 
The reasons for this are explained in the next section. 



The Requirements of "Perfect Implementation " 

Why is implementation so difficult? Why do policies never seem to work 
right? Mainly because the ideal state of "perfect implementation" is 
virtually unattainable in practice. Here, "perfect implementation" is 
used not as something to be achieved in JTPA (or any other program), but 
as a concept for understanding why policies do not work as intended. The 
requirements of "perfect implementation"— that is, the conditions needed 
for a policy to work right— are listed below. [6] 

• CIRCUMSTANCES EXTERNAL TO THE IMPLEMENTING AGENCY DO 
NOT IMPOSE CRIPPLING CONSTRAINTS. Some obstacles to 
implementation are outside the control of adminis- 
trators . For JTPA , these i ncl ude downturn i n the 
national economy, or changes in political support for 
JTPA in Congress. When these occur, even good policies 
can fail. 
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ADEQUATE TIME AND SUFFICIENT RESOURCES ARE MADE 
AVAILABLE TO THE PROGRAM. Programs and policies that 
are physically and politically feasible may fail either 
because too much is expected too soon or because 
insufficient resources are authorized to carry them 
out. 

THE REQUIRED COMBINATION OF RESOURCES IS ACTUALLY 
AVAILABLE. Not only must adequate resources be 
available, they must be available at each stage of the 
implementation process. The initial flow of funding 
may give life to a program, but there is no guarantee 
that the flow will continue throughout the planned 
duration of the program. Furthermore, even if funding 
is achieved, there is no guarantee that funds can be 
converted into materials, labor resources, or office 
space within the time-scale of the program, 

THE PROGRAM IS BASED UPON A VALID THEORY OF CAUSE AND 
EFFECT. Policies are sometimes ineffective not because 
they are poorly implemented, but because they are bad 
policies. A policy may be based on an inadequate 
understanding of the problem to be solved, its causes 
and cure; or of an opportunity, its nature, and what is 
needed to exploit it. That is, the typical reasoning 
of the policymaker is along the lines of, "If we do X 
now, then Y will result." Every policy is based on 
such a theory of cause and effect (normally unstated in 
practice), and, if the policy fails, it may be the 
underlyi ng theory that i s at f aul t rather than the 
implementation of the policy. An example is the 
failure of a program in the late 1960s aimed at 
creating employment for members of minority groups in 
Oakland, California. [9] The program provided funds 
for publ i c works and for maki ng 1 oans avai 1 abl e to 
private firms, in hopes that (a) new business 
opportunities would be created, (b) which would lead to 
new jobs, and (c) which would be filled by minorities. 
Despite much good will , the program achieved very 
1 i ttl e. Few 1 oans were taken up by f i rms and some 
proposals for public works came to nothing. Even when 
funds were spent as intended, the net increase in 
minority employment was small. The program failed 
mainly because it was based upon an inadequate theory 
of job creation. 

THE RELATIONSHIP BETWEEN CAUSE AND EFFECT IS DIRECT AND 
THERE ARE FEW (IF ANY) INTERVENING LINKS. Some 
policies have complex underlying theories. Instead of 
"if X then Y," the policy may have a longer chain, such 
as "if X then Y, and if Y then Z." The more links in 
the chain, the greater the risk that some of them will 
prove to be either poorly conceived or badly executed. 
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In other words, instead of taking the direct path of paying the employer 
a subsidy on wages after they had hired minority personnel, the program 
expanded business capital on the promise that business would later hire 
^ the right people. 

ERIC 



Any social program worth having a governmental policy about 
at all is likely to be a serious and complicated problem and 
therefore not amendable to easy solution or even amelio- 
rati on . To state the proposi ti on a 1 i ttl e di f f erently , 
government gets the dregs of our social problems to solve. 
The easy ones have been dealt with already by individuals 
looking after their own interests, by families and other 
social groups looking after their members, and by economic 
units profitably exchanging labor, goods, and money in the 
marketplace. Government these days picks up the problems of 
physically and psychologically dependent persons, of social 
groups who have historically been stripped of the capacity 
to protect themselves from other social groups, and of 
economic units that cannot make it in the competition of the 
marketplace. To a certain extent these problems have been 
created as by-products of the successes of governmental 
solutions to other problems, as in black teenage unemploy- 
ment being a by-product of minimum-wage legislation. In any 
case, they are simply "left over" by the nongovernmental 
(private and customary) and governmental processes that have 
on the whole done quite well at preventing an even larger 
incidence of problems. When government today turns to 
solving the "leftover" problems, it is rapidly discovered 
that no one knows quite what to do. Overwhelming political 
pressures often dictate, however, that "something" must be 
tried. Assertions that the effort is very likely to be 
futile (or worse) are discounted as weak apologies for the 
status quo. Clearly, we would all be better off. of course, 
if government simply stayed away from problems it could not 
solve. Alas, if only we could know, before trying and 
failing, what they were! 



In summary, the JTPA service delivery system has several characteristics 
that increase the difficulty of conducting pt-ocess evaluations of state 
policies. These characteristics include multiple organizations, 
decentralization, local autonomy, and the limits of performance control 
systems. Methods for conducting state process evaluations must take 
these characteristics into account, recognizing that a single policy may 
affect different SDAs in different ways. The methods presented in the 
next section attempt to achieve this end. Thus, they may not be perfect, 
but they are practical for the real world in which JTPA administrators 
work. 
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Solutions (Policies) Are Proposed 
Alternative Policies Are Compared 
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Policy implementation 
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INTRODUCTION 

Many process evaluations of policies and public programs have been 
conducted in the past. Usually, they are case histories of the policy's 
ongin, its development through the decision-making process, and its 
consequences once implemented. The aim is to gain an understanding of 
what has occurred in the past with the hope that it will serve as a guide 
to the future. The Grinker-Wal ker (1985) reports cited in this volume 
are examples of this type of analysis. 

This historical approach, however, may have little utility for state (and 
local) JTPA administrators for feveral reasons. First, it is not 
future-oriented. That is, it does not necessarily help state admini- 
strators design successful policies either to prevent potential problems 
from occurring or to solve existing ones (or at least reduce their 
severity). Second, it does not necessarily help administrators overcome 
the system constraints mentioned earlier. For example, under local 
autonomy, SDAs and subcontractors can stymie state efforts to trace a 
policy's influence on local operations following implementation. Third, 
and perhaps most important, is the so-called "point-of-view" issue. [1] 

All parties in the implementation process— pol icy-making or otherwise-- 
are involved, in some degree, both in trying to control others and in 
trying to avoid being controlled by ti.am. That is, all parties have 
implementation problems, and what some may perceive as "solutions" others 
will see merely as aggravations, or worse. In JTPA, decentralization, 
multiple organizations and local autonomy can foster conflict in state- 
SDA relations. An historic approach to policy evaluation offers little 
guidance to JTPA officials in working through their differences to build 
successful policies. 

For the above reasons, a case-history approach is rejected and a future- 
oriented policy-making and implementation process analysis is offered in 
its place. This "state process evaluation model" consists of three 
sequential activities: policy design, policy decision, and policy 
implementation, as shown on the left-side of Exhibit 2. 

The process in Exhibit 2 has an hour-glass shape. Policy design, the top 
half of the hour-glass, begins on a broad note with the identification of 
an actual or potential problem. Many actors become involved in the 
process as alternative solutions, or policies, are proposed. These are 
debated until, through one mechanism or another, one policy is chosen for 
implementation. 2/ 



2/ Conceivably, a decision could be made not to implement any policy. 
The process would end at this point, unless new alternatives were 
presented for debate. 



161 

17 



EXHIBIT 2: THE STATE PROCESS EVALUATION MODEL 
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Thus, although the process begins with sundry broad issues, 
narrowing, ultimately culminating in a decision to implement 
policy. 

The bottom half of the hour-glass, on the other hand, is self 
After being concentrated in one place (i.e., the decision-r 
policy spreads throughout the service delivery system. Depen 
specific policy, a bit of its implementation process may go 
intake interview; in every piC meeting where the SDA budget is 
in every subcontractor where a counselor picks up a telephone 
a client referral. The focus of policy implementation is ex, 
policy's influence on SDA (and subcontractor) operations 
constraints discussed earl ier--that is, making sure the policy 
as its creators intended. 

Organizational and communication supports must be present in 
for states (or any other JTPA agency) to implement the mod< 
fully. Although Congress divided responsibility for JTPA ami 
SDAs and subcontractors to achieve certain program bene 
division also has undesired consequences. Specifically, it c 
restrict the flow of information from one agency to another, 
that is transmitted between agencies can be highly filtered, 
is highlighted; bad news is blocked. Few state administrat 
comprehensive awareness and understandino of what goes on a1 
level and the problems service providers face. When problems 
they often erupt under crisis conditions fueled by inxense 
pressures with parties or interest groups demanding an immedia 
(or attempting to block one). Under these circumstances, stc 
tempted to specify, implement and enforce policy in a "top-( 
that ignores local concerns. Of course, problems do get sol^ 
way, hut oftentimes only a few interest groups are satisfie 
results, and the political process has hurt future 
subcontractor relations. In short, while setting up a general 
Congress has failed to specify a mechanism to help states 
subcontractors work together to solve their common problems 
course, is not necessarily unreasonable; Congress cannot ant' 
great variation that exists among the states. 

States, SDAs and subcontractors can create their own me 
establishing forums and communication networks that bring a' 
parties together on a regular basis to air their common 
problems, exchange information, and reach workable solutions ai 
the parties of interest. Examples of such forums are listed I 
of these already exist in several states: 

• The SJTCC can serve as a forum to debate and solve SDA p< 
issues; 



• Special task forces (made-up of states SDA and subcontractor 
representatives) can be created to study a problem, evaluate 
al ternati ve sol uti ons , and recommend a pol icy for 
implementation: 

• Subcontractor conferences can be convened where subcontractors 
n^aet regularly with SDAs to raise and solve policy issues; 

• State-SDA-subcontractor-coordinating agency conft^ances can be 
convened, which bring all parties in JTPA together to share 
information, raise problems anc* debate solutions; and, 

• An independent agency can be established to solve 
"irreconcilable*' policy conflicts among states, SDAs and 
subcontractors. 

Contacts established in the forums can also lead to inforiual 
communication networks traversing all organizatioi^s in the system. 3/ 

Implementation of the state process model within such forums would 
proceed as follow^. (Please refer to the right-hand elements of 
Exhibit 2): 

PROBLEMS ARE DEFI^^ED: State, SDA and subcontractor representatives 
(hereafter referred to as *'nembers") debate issues to reach common 
definitions of mutual oroblems. Causes of the problem are 
identified; 

SOMJTIOHS ARE PROPOSED: Members propose alternative policies 
(agreeable to them) that deal with the causes to solve the problem. 
Mechanisms for implementing each policy a^e aUo proposed: 

ALTERNATIVE POLICIES ARE COMPARED: Members compare the merits of 
the alternative pol icies recommending one (or more) for implemen- 
tation. 4/ Because the members themselves have studi^pid the problem, 
debated cheir own alternative solutions, and made their own 
recommendations, tria recommended policy will likely be acceptable to 
most members and JTPA agencies; 



3/ Liaison staff responsible for interagercy affairs also create 
informal communication ne^work^. through their work. Mowever , 
1 iaison staff are often part of the perfoririance control system, 
responsible for supplementing MIS data with other "soft" information 
about what's going on in another agency. This mo^^itorinq/compliance 
role may inhibit and/or distort communication flows. 

4/ States may wish to delegate decision-making authority to the forum. 
In this case, the members would select a policy for implementation, 
not just recommend one. 



A CHOICE IS MADE: The polv:y-maker decides to implement the policy, 
or refer it back to the forum for further discussion. However, the 
probability of this happening can be reduced if the policy-maker is 
also a member of the forum- and, 

THE POLICY IS IMPLEMENTED: Members and the policy-maker monitor 
policy implementation. Prior to implementation, members have 
determined how the policy should work (i.e., cause-effect) in the 
field, and what local managers should do to implement it properly. 
Local activities in these specific areas are monitored to determine 
If the policy is working as intended and whether the problem is 
disappearing. If the policy is not working as intended, it is 
revised or replaced with an alternative. 

In this process, a policy's ultimate success is determined largely BEFORE 
it is ever implemented. Here, success is contingent on having a working 
membership composed of state, SDA and subcontractor officials that is 
recognized and respected as the system's policy authority. This process 
can hopefully defuse and solve problems before they erupt as full crises. 

In the sections that follow, methods for implementing each part of the 
model are presented. 



THE PROBLEM IS DEFINED 

Problems requiring policy intervention are not defined by operation 
manuals, MIS computer systems or in employment and training handbooks. 
For the most part, what defines a problem depends greatly on what FACTS 
are known about the system, what VALUES are being applied, and WHO has 
placed the problem on the agenda from what level and for what reasons. 
Let's look at each of these factors. 

In any given state there is much information describing "what goes on" in 
JTPA.^^ These facts may be either "hard" (such as MIS performance data) or 
"soft" (such as gossip, hearsay, or speculation). Examples of such facts 
include 1) high SDA placement rates, 2) the number of jobs created 
through an economic development program developed in a SDA, 3) a high 
level of creaming across SDAs, 4) a low level of coordination among local 
agencies, or 5) the joy of a youth finding his or her first job. By 
themselves, facts are merely descriptive of JTPA's operations. But 
through pol it;cs--the system we have for attaching values to facts- 
judgments about what is good and bad with JTPA become possible. [6, 15] 
These judgments cannot be PROVED correct or incorrect; they can only be 
APPROVED as right or condemned as wrong by the exercise of another value 
judgment. Thus, if we used congressional intent (i.e.. Congress' values) 
to judge facts 1-5 above, facts 1, 2 and 5 could be classifed as "good" 
while facts 3 and 4 could be classified as "bad." 

But does this make everything in the "bad" category a problem requiring 
solrtion? No, not necessarily. Through the political process, someone 
with sufficient power must place the problem on the agenda before efforts 
are directed at solving it. Fo:- example, PICs may repeatedly petition 
for more state-spontored economic development programs, but it is not 



until the Governor declares a poor economy as being a statewide issue 
that such programs emerge. Definitions of problems also may change over 
time as more facts emerge, or values change, or power shifts from those 
who perceive a problem to those who do not. [6] 

In summary, there are no hard-and-fart rules for deciding what is or is 
not a problem in JTPA. These will always be political decisions that 
change over time. Thus, what one state defines as a problem may go 
unrecognized as such in another state. 

Once a problem is defined, the next step is to collect more information 
about it. Specifically, the problem can be defined along the following 
dimensions: 

• SCALE AND INTENSITY: How widespread is the problem? How serious is 
the problem for those affected? 

• INCIDENCE: Is the problem distributed equally among all those 
affected? If incidence is not uniform, future policies will have to 
discriminate among recipients , and a simple ^'package" may not 
suffice. 

• CHARACTERISTICS (OF THOSE AFFECTED): Is the problem particularly 
serious for some groups, such as youth? 

• TARGET ABILITY OF PROGRAMS: What data do we need to direct resources 
where they will do the most good? 

• ATTRACTION: What attracts people into programs? Problems often 
result in programs to solve them. Will people change their behavior 
in order to qualify for the program and receive its benefits? This 
can artifically increase the scope of the problem. 

• RATE OF CHANGE: How is the problem changing? For example, are 
unemployment rates rising or falling sharply? 

• UNCERTAINTY: How will the problem develop in the future? Will the 
problem solve itself? What are the "most optimistic" and "most 
pessimistic" scenarios? 

Answers to these questions are used to develop a precise definition of 
the problem and provide the basis for identifying its causes. To 
illustrate how this is done, I will first use Pressman and Wildavsky*s 
study of a jobs program implemented in Oakland, California, during the 
late 1960s. [9j Although the program was judged a failure, it offers 
several lessons for this discussion. 

On the surface, the problem can be simply stated. Oakland had a large 
minority nopulation; its unemployment rate was twice the national average 
(8.4 percent vs. 4.1 percent national average); and racial tensions were 
high. Most observers felt that Oakland was a tinderbox--the next War'-;, 
ready to blow. Action was needed to prevent a serious problem from 




becoming worse. In its simplest terms, the problem was defined as a lack 
of jobs for the unemployed and underemployed. It was felt that immediate 
action was needed to prevent a future riot in Oakland. 5/ 

The causes of Oakland's minority unemployment problems were never fully 
understood by those involved with the program. However, Pressman and 
Wildavsky offer their own interpretation. [9, p. 152-153] 

...The urban ghetto is often within two blocks of some 
of the highest value of real estate in the nation. 
Industry and commerce thrive around it so that the 
unemployed can sit in their doors and watch the "fat 
cats" go to work. Typically, this distressed neighbor- 
hood 1 i es i n the heart of a growi ng , di versi fy i ng 
region that is by no means "outside the mainstream of 
the Nation's prosperity" but is at its very heart... 

. . . The East Bay i s an area of rapi d growth wi th a 
diverse economic base, a high per capita income, and so 
forth, but in which, because of certain charactersi tics 
of the postindustrial economy and deficiencies in the 
labor market, many residents remain chronical ly un- 
employed, even during periods of high aggregate 
ciemand. . .The problem is... one of rigging a strong and 
well-developed metropolitan economy so as to reduce the 
disparity between the location and characterstics of 
jobs available and the location and skills of the 
distressed population. 

In short, if the problem is either poorly conceived or does not exist, in 
fact, even an "excellent" solution will not work in practice. 

The following example shows how JTPA officials might go about defining a 
problem. In one state all SDAs have failed to achieve their respective 
performance standards for the number of yjuth dropouts with positive 
outcomes. The state, with the help of SDAs and subcontractors, has 
agreed to study the problem and formulate an appropriate policy response. 
To begin, two possible sources of the problem are proposed, and within 
each source, alternative causes of the problem are voiced, as shown 
below. 

SOURCE OF PROBLEM POTENTIAL CAUSES IN EACH SOURCE 

ENVIRONMENT 1. The number of youth dropouts in the SDA's 

population is lower than expected. 

2. Another state program is already providing 
services to youth dropouts. 



5/ The person perhaps most responsible for labeling these conditions as 
problems was Eugene Foley, Assistant Secretary of Commerce in charge 
of the Economic Developftient Administration. 



SOURCE OF PROBLEM POTENTIAL CAUSES IN EACH SOURCE 



SDA/SUBCONTRACTOR 3. An adequate number of youth dropouts exists, but 

outreach/intake services have not reached them. 

4. Enough dropouts apply and enroll in JTPA, but 
too many also dropout during training. 



To determine which, if any, of these potential causes is correct, 
information about the problem's dimensions is collected. From these 
data, the following facts became important to those studying the problem: 

1. Information from the state's public education authority reveal that 
substantial numbers of youth drop out of high school in a maioritv 
of the SDAs. 



The authority also provides information showing that several high 
schools with high dropout rates operate their own outreach programs 
to get youth back in school. JTPA services are rarely coordinated 
with these efforts across SDAs. 

SDA intake figures clearly indicate that the number of dropouts 
applying for services is usually lower than the SDAs' required 
performanct standards. 



These facts indicate that both the environment and the SDA/subcontractor 
are sources of the problem. The environment is a source because another 
program is already serving this target group. The SDA/subcontractor is 
also a source because JTPA programs fail to attract dropouts who are not 
already enrolled in programs operated by some high schools. 

Now that the problem is defined and its causes understood, the 
policy-maker(s) can begin thinking about what must be done to solve the 
problem. That is, the policy-maker needs to specify a set of conditions, 
or "objectives," that, when satisfied, would indicate that the problem is 
solved. In this example, five policy objectives a/e proposed, each 
categorized along five dimensions, as shown in Exhibit 3. Of course, 
other objectives are still possible. For example, one objective might be 
that a policy require little management to implement. However, the 
dimensions listed here are major concerns in most policy debates. 



EXHIBIT 3 

POLICY OBJECTIVES FOR YOUTH DROPOUT PROBLEM 



DIMENSION 



OBJECTIVE 



Effectiveness 1. 



Efficiency 



Organizational Fit 



Equity 4. 



Uncertainty 5. 



Three-fourths of the SDAs 
performance standards for 
within two quarters after 
implemented. 



satisfy their 
youth dropouts 
the policy is 



The average positive outcome cost for youth 
dropouts increases by no more than five 
percent. 

The policy does not require SDAs and 
subcontractors to greatly reorganize their 
staff to accommodate the new policy. 

Tha distribution of racial /ethnic groups among 
served dropouts matches the distribution in 
the SDA's youth dropout population. 



The policy can adapt 
in dropout rates 
population- 



to unpredictable changes 
in the • SDA's youth 
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In summary, at this point in the process a clear (and hopefully accurate) 
definition of the problem has been established, and the causes of the 
problem are understood by most members engaged in the policy-making 
process. Furthermore, a set of objectives has been established that, 
when achieved, will indicate that the problem has been solved. As we 
shall see later, the objectives will become invaluable later in comparing 
alternative policies and monitoring their influence on SDA/subcontractor 
operations during policy implementation. Now, let's move on to the next 
step. 



SOLUTIONS (POLICIES) ARE PROPOSED 

There are usually several ways of solving any given problem, some more 
effective than others. In this step, our interests are: 1) proposing 
alternative policies, and (2) explaining how each solves tlis problem^ 
This explanation has two components. First, how a policy will deal with 
the causes to solve the problem must be explained explicitly. That is, 
at this stage, each proposed policy is a hypothesis--or an educated 
guess—that it, when implemented, will nullify the causes of the problem 
and, hopefully, will do so without undesirable financial , political , 
social or other consequences. The second component is that the policy be 
a workable solution of the problem. Although a policy may address the 
causes of a problem well, the policy is of little value if it cannot be 
i mpl emented i n the field. Let ' s return to the Oakl and unempl oyment 
problem to illustrate this step. 

Even though the sponsors lacked a clear definition of Oakland's 
unemployment problems, they had a clear idea of how to solve them. Their 
basic approach was that a massive injection of federal money could create 
permanent jobs for the hard-core unemployed. Their central hypothesis 
was that a program of grants and loans to businesses would create 
3,000 new jobs in the city. They saw the business grants/1 oai*«s as a way 
of involving private sector firms of various sizes in the effort to 
provide jobs for the unemployed. 

However, their approach had some fundamental flaws. The underlying 
question, raised by one orflci?!, was, "Aren't we skirting the problem? 
...how do you connect with the long-term unemployed? We still have not 
heard a formula on guaranteeing that the right people get the right 
jobs." [9:24] The following explains in more detail why this approach 
failed. [9:149] 

...They did understand that it was insufficient merely to 
create economic growth; they knew they had to seek 
assurances that new jobs would go to the minority people in 
whose i nterest the federal money was presumably bei ng 
expended. But, their desire to move with the greatest 
possible speed persuaded them to devise a program based on 
promi ses rather than performance. Thei r i nnovati on , they 
thought, was funnel ing money into a city rapidly on 
condition that it be used to generate the right kind of 
employment. Their creativity consisted in moving money and 
devising mechanisms of compliance after it had been spent. 
No one asked them whether it might have been better to give 
firms incentives to hire minority personnel by paying part 
of the cost after the men had beey >^i^ed. 

ERIC 



In other words, instead of taking the direct path of paying the employer 
a subsidy on wages after they had hired minority personnel, the program 
expanded business capital on the prornise that business would later hire 
the right people. 

Another fault of the Oakland project was that its implementation plan was 
unworkable. In fact, there was no implementation plan. [9:143] 

From the outset, the emphasis was on designing the program, 
obtaining initial agreement at the local level, and 
committing the funds. All this was done quickly, with 
fanfare and enthusiasm. . .The latter steps of implementation 
were felt to ba "technical questions" that would resolve 
themselves if the initial agreements were negotiated and 
commitments were made. But the years have shown how those 
seemingly routine questions of implementation were the rocks 
on which the program eventually floundered. 

The lesson to be learned is that the design of policy and how that policy 
will be implemented must be considered jointly, if the policy is to be a 
success. 

One way of doing this is to create alternative strategies for 
implementing each proposed policy. Each strategy would describe 
administrative requirements for implementing the pol icy, anticipating 
possible pitfalls, political pressures, resource limits and other 
possible barriers. 6/ In general, strategies will have better chances of 
success if: (1) they implement a policy based on a sound theory, and 
(2) implementation remains simple and requires little management. [1,6,9] 
Examples of such implementation strategies include the following: 

• Targeting policy directly on the problem; 

• Minimizing the number of steps and decisions needed to 
implement the policy; 

• Paying as much attention to the creation of organizational 
machinery for implementing a policy as for designing one; 

• Providing the implementing agency with additional resources to 
carry out the policy; ana, 

• Inserting performance incentives into the implementation 
process (where an incentive is defined as the promise of a 
reward contingent on demonstration of certain performance, with 
the amount of the reward proportionate to the degree of 
performance). 



6/ Many, perhaps most, constraints remain hidden in the planning stage 
and are only r! « scovered in the implementation process. Further, 
feasibil ity conditions keep changing over time: old constraints 
disappear or are overcome (e.g., through learning), while new ones 
emerge. [9] 



Now, let^s illustrate how alternative policies are developed through the 
youth dropout problem presented earl ier. The task of creating an 
alternative may be best described as "structured brainstorming." It is 
"brainstorming" because it basically involves thinking about new and 
different ways of solving a problem. In general, as the number of 
persons involved in this activity increases, so will the number of 
alternative policies. It is critical that state, SDA and subcontractor 
representatives be involved in this stage (through the various forums 
proposed earlier) because, due to their positions in the system, a sub- 
contractor may have quite different ideas about how to solve a problem 
than a state official. 

This brainstorming is "structured," however, because it must develop 
alternative policies that take into account the definition of the 
problem, its causes, and the specific objectives that a future policy 
should satisfy. That is, a policy should clearly address the problem, 
nullify its causes and satisfy most (if not all) of the objectives to 
receive serious consideration. The multitude of facts collected in the 
previous step can often help members to think of policies that meet these 
concerns. 

For the youth dropout problem, three policies are proposed. Exhibit 4 
contains a brief description of each policy and its underlying theory 
(i.e., how it solves the problem), implementation strategy, and ability 
to satisfy objectives. By constructing such a table, all members of the 
policy-making process can reach a common understanding of exactly what is 
being proposed to solve the problem and how it should work. In the next 
step these alternative policies are compared. 

ALTERNATIVE POLICIES ARE COMPARED 

In this step the alternative policies are compared, and one or more are 
recommended for implementation. 7/ Returning to Exhibit 2, the key point 
here is that policy comparisons are just one step in a much larger stream 
of activity designed to solve pressing problems. Those making the 
comparisons are usually aware that: (1) a policy decision will be made 
at a certain time; (2) action will be taken at that lime; and (3) such a 
decision cannot be based on information that is available later. For the 
task of making policy comparisons, this means partial information avail- 
able now is better than complete information after implementation 
begins. [2] But, the mere availability of partial information does not 
guarantee that it will be used in making policy comparisons. If the 
information does not help policy-makers predict the success of alter- 
native policies, it will likely be ignored. Moreover, intense political 
pressures advocating one policy over all competitors may virtually 
eliminate any formal comparisons of alternative policies from the 
process. 



7/ This, of course, assumes that more than one policy was proposed in 
the previous- step. As mentioned earlier, if the members of a forum 
cannot reach agreement, the process ends or starts all over. If the 
decision-maker is also a member of the group, then the recom- 
mendation is also the decision (i.e., the next step in the process). 



EXHIBIT 4 
ALTERNATIVE POLICIES 



DESCRIPTION 



HOW IT WORKS 



IMPLEMENTATION PLAN DOES IT MEET OBJECTIVES 



Ise state six percent 
unds to provide SDAs/ 
ubcontractors with a 
bonus" for each 
ositive outcome for 
outh dropouts. 



Payment on performance 
creates incentives to 
serve more dropouts. 



Provide bonuses after 
SDA gives state proof 
of performance. 



oordinate JTPA services 
ith high school 
rograms. 



Most dropouts already 
in school district 
programs. Therefore, 
school districts sub-- 
contract with SDA/ 
subcontractor for 
services. High school 
responsible for co- 
ordinating program. 



School districts mainly 
responsible for imple- 
mentation. SDA 
appoints one liaison 
staff to coordinate 
JTPA services with 
district high schools. 



Fits well into SDA and subcontractor 
operations. Average outcome costs 
can be met, but amount of bonus 
may be too low for 3/4 of all SDAs 
to meet standard with all racial/ 
ethnic groups represented. 
Satisfies uncertainty. 



With access to dropouts increased, 
most SDAs should meet standards. 
Most racial groups should be 
represented based on past high 
school records. Fits well into 
SDA's subcontracting operation, 
but SDAs wary of district control 
of SDA services. May not meet 5% 
cost ceiling, but can drop school 
district subcontract if youth 
dropout problem declines. 



rovide technical 
ssi stance to SJAs/ 
ubcontractors to 
evelop and operate 
ustomized intake 
srvices for dropouts. 



Customized outreach 
will be able to 
locate and enroll 
dropouts not picked- 
up by high schools. 



Hire consultants to 
train SDA/Subcontractor 
outreach staff. Show 
staff how to integrate 
dropout outreach 
services into on- 
going youth outreach 
services. 



Effectiveness of new outreach 
unknown, so 3/4 of SDAs may not 
satisfy standards. Satisfies 
cost limits and equity require- 
ment. Fits well into current 
outreach efforts. Can be stopped 
if youth dropout rate in 
population falls. 
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In short, the task oT comparing alternative policies can range from the 
very simple (e.g., a hallway discussion between two power figures in the 
policy-making process) to the very complex (e.g., a cost-benefit 
evaluation of each alternative). The aim here is to reach a middle 
ground: to present a practical and simple method for comparing 
alternative policies that can be implemented in a relatively short amount 
of time. 



The method chosen is the POLICY MATRIX, shown below. [5,12] The matrix 
has three parts: (1) the desired ends/objectives— what the policy is 
supposed to achieve; (2) the policies, or ways of accomplishing the ends; 
and (3) outcomes, vr the ability of a specific policy to achieve a 
specific objective. The objectives are the rows; the policies are the 
columns: and the outcomes are the cells. As a whole, the matrix presents 
an overview of the strengths and weaknesses of each policy which can be 
studied quickly to arrive at an informed policy recommendation or 
decision. 



EXHIBIT 5 



POLICY MATRIX FOR THE YOUTH DROPOUT PROBLEM 



POLICIES 



^ 1 ' 



2 



OBJECTIVES 3 




Before policies can be compared, two tasks must be performed: (1) the 
imp. -tance of each objective must be decided; and (2) the predicted 
outi.omes of each policy for each objective must be estimated. Each task 
is illustrated using the youth dropout problem introduced earlier in this 
section. 

Exhibit 3 contains the fiv:. policy objectives for the youth dropout 
problem. It is unlikely that each objective has equal importance to 
policy-makers. Their relative importance--which is always a value 
judgment— cnn be measured by having policy-makers distribute 100 points 
across the five objectives. The more important the objective, the more 
points it should be assigned. This activity can be conducted in a group 
setting, so state, SDA and subcontractor officials can reach common 



agreement on how the points should be distributed. [7,12] In o 
example, the following distribution of points was decided: 



OBJECTIVE 


POINTS 


1/Effectiveness 


35 


2/Eff iciency 


20 


3/Organizational fit 


20 


4/Equity 


15 


5/Uncertainty 


_1_0 




100 



The next task is to predict the outcomes of each policy (as described in 
Exhibit 4) for each objective. If policy-makers so desire, a consid- 
erable amount of effort may be devoted to calculating predicted outcomes 
For example. Appendix B provides an excellent example of how con- 
effectiveness analysis can be applied to predict outcomes of job training 
programs. Other popular techniques, such as cost-benefit analysis, can 
also be used. [13] As mentioned earlier, however, policy-makers may not 
have the time, money, skills or inclination to use these techniques, 
particularly if political pressures are demanding an immediate solution* 
In these cases, the information collected in the previous steps may oe 
sufficient to make educated guesses about a policy's ability to achieve a 
specific objective. However, if conditions permit, collecting additional 
information for evaluating policy outcomes is encouraged. 8/ 

There are different ways of measuring how well a policy satisfies an 
objective. Some examples are listed below: 

DEFINITION 

The alternative policies are ranked 
according to their ability to meet an 
objective. 

Outcomes for an objective .Tiay be a 
verbal scale, such as poor, fair, 
good, excellent. 



ME ASURE 
Rank 

Verbal 



8/ Different types of information can be collected. For example, SDAs 
and subcontractors can be surveyed to determine their opinions about 
each alternative policy. The problems of attracting dropouts to 
JTPA programs can be studied in greater depth. The causes of 
dropping out in the first place can be studied, with an eye toward 
developing a joint school district- JTPA program aimed at preventing 
dropouts. Or each alternative could be implemented on a trial basis 
in different SDAs to see which one works best. 



MEASURE 



DEFINITION 



Points 

Monetary 
Time 



Instead of using a vertial score, 
points can be assigned. For example, 
100 points means a policy satibfies an 
objective completely, while 0 points 
means the opposite extreme. 

How much does it cost for the policy 
to achieve an objective? 

How long will it take for a policy to 
achieve an objective? 



Deviation from Objective Points are assigned based on how well 

the policy satisfies or fails to 
satisfy an objective. For example, 
25 points can be assigned if the 
policy greatly exceeds the objective, 
0 points if it satisfies the objective 
exactly, -25 points if it completely 
fails to meet the objective. 

These ave just a few of several, alternative measures that could be used. 
Once a measure is chosen for each objective, outcome scores can be 
calculated or assigned and inserted into the matrix, as shown below for 
our youth dropout example. The objectives correspond to those in 
Exhibit 3; the policies to Exhibit 4. 



EXHIBIT 6: YOUTH DROPOUT POLICY MATRIX WITH MIXED MEASURES 





Priority 




POLICY 






Objective 


Points 


A 


B 


C 




1 


35 


0 


10 


-5 


Deviations 


2 


20 


$39 


$55 


$45 


Cost (per youth) 


3 


20 


Good 


Fair 


Good 


Verbal 


4 


15 


3 


1 


2 


Rank 


5 


10 


80 


40 


60 


Points 



Exhibit 6 contains a variety of measures across the objectives-- 
deviations, cost, verbal, rank and points. With the objectives, 
policies, priority points (reflecting importance of the objectives) and 
outcomes summarized in one place, policy-makers can quickly compare the 
merits of one policy versus another along a number of dimensions. Seeing 
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the tradeoffs in one place often forces policy-makers to rethink their 
judgments from previous steps. It is critical to note, however, that the 
Exhibit 6, by itself, does not indicate which policy is superior to all 
others. 9/ This decision always remains the responsibility of the 
pol icy-maker. 

In the youth (dropout example, the importance given to Objective 1 (by the 
35 priority points) eliminates Policy C from further consideration (due 
to Its negative score). The choice between Policies A and B is difficult 
because of their relatively similar scores. The final recommendation in 
this case is Policy A, due to its low cost (Objective 2). Next, the 
recommendation is forr^sally considered by the policy decision-maker(s). 

A CHOICE IS MADE 

At this point in Ihe process (in Exhibit 2) an individual or group having 
the authority makes a decision regarding the policy. The decision is 
usually either tr^ implement the policy as recommended, or to implement a 
revised version of the policy, or to choose a different policy for 
implementation, or to not implement any policy. In the last case the 
process eit.'ier ends or starts over again as policy-makers go back to the 
t, awing boards tr develop new policies for consideration. 

If the decision-maki ; nas been largely removed from the policy-making 
process, she or he may be unaware of the many tradeoffs, judgments and 
other decisions made in the previous three steps. Therefore, it is 
important that some of the history of the process be conveyed' to the 
decision-maker in some form, such as a report, a memo or even a telephone 
conversation. 

In our youth dropout problem, the decision-maker concurs with the 
recommendation and Policy A, incentive payments, is approved for 
implementation. As we shall see in the next section, the information 
collected in the pol icy-maki.ig process will play a key role in monitoring 
policy implementation in the next stage. 



POLICY IMPLEMENTATIO'" 

To reinforce a message stated earlier, every policy has two parts, an 
underlying theory and an implementation strategy. A decision to 
implement a policy is really an educated guess that the policy's theory 
and implementation strategy will produce desired outcomes. Because no 
one can foresee the future with complete accuracy, policy-makers need to 
monitor outcomes to verify that the policy is working as intended and 
that it has few undesirable side effects. 



9/ If all the outcome scores are quantified and most are commensurate, 
an overall performar'-e score for each policy can be calculated. 
[5,7,12] While this score measures which policy is best, policy- 
makers are by no means obligated to choose that policy for 
implementation. 
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If a policy is not working right, it may be because its underlying theory 
is incorrect, or its implementation str^.tegy is unworkable, or both. If 
the reasons for policy failure are understood, policy and decision-makers 
can possibly revise, terminate or otherwise alter it as deemed necessary. 

Information from the local level must be collected in order to make these 
judgments. However, (state) officials responsible for monitoring a 
policy will not be able to monitor everything that goes on at the local 
level. The geographic separation of the state and local areas makes such 
an approach unworkable, plus it interferes with the normal operation of 
the state's performance control system. Further, even if a state wanted 
to monitor comprehensively, it would likely not have the resources to do 
so in all SDAs. Moreover, state attempts to collect comprehensive 
information on SDA operations will likely be perceived as unwarranted 
harassment by local officials and will be resisted. The proper tactic, 
therefore, is to restrict monitoring to the policy objectives developed 
previously. If the objectives capture both th(^ underlying theory and the 
implementation strategy of the policy, the objectives will serve as a 
useful means of focusing the monitoring effort. In this case, the key 
monitoring questions become: 

• Has the policy achieved its objectives? 

• Has the policy produced unanticipated outcomes, or side effects? 

• Why did these outcomes occur? 

To answer these questions, indicators that measure the policy's outcomes 
at the local level must be developed. The choice of indicators depends 
greatly on the policy's underlying theories, implementation strategy, and 
objectives. Let's use Policy A (Exhibit 4) to illustrate how this is 
done. 

Policy A's underlying theory is simple. By providing a reward, state 
bonus money, contingent on proof of performance, local authorities will 
have increased incentives to serve youth dropouts, which--in some un- 
specified way— will solve the youth dropout problem. Instead of evading 
state scrutiny, SDAs might seek out such scrutiny to receive their 
bonuses. Its implementation strategy, distribut-^ng funds based on proof 
of performance, also sounds simple, at least from a state perspective. 
But either the policy's underlying theory or implementation strategy (or 
both) could break down. The underlying theory might fail if the bonus is 
not big enough to motivate local officials to serve more youth dropouts. 
Or the theory might fail if local authorities only serve dropouts in one 
racial group and ignore the rest (for example, because they are harder to 
find). The implementation strrtegy might fail if SDAs are required to 
submit considerable paperv/ork to prove their performance. Or it might 
fail if an administrative SDA Allocates little of the bonus money to 
subcontractors, who are actually doing the work. The policy might also 
have undesirable side effects, ^ ich as fewer services for youth who are 
not dropouts. The key point, len, is (1) to develop indicators that 
capture a policy's underlying theory, implementation strategy and 
possible side-effects, and (2) to relate the indicators to the policy's 
objectives. An example follows for the Policy A objectives. 
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POLICY 
OBJECTIVES 

1. Three-fourths of SDAs 
satisfy youth dropout 
performance standards. 



2. Average cost for dropouts 
increases by no more than 
five percent. 

3. SDAs/subcontractors must not 
reorganize service delivery 
to accommodate policy. 



4. The distribution of racial/ 
ethnic groups among served 
dropouts matches the dis- 
tribution in the SDA's 
youth dropout population. 

5. Policy can adapt to changes 
in dropout rates in SDA*s 
youth population. 



EXAMPLE 
INDICATORS 

Youth dropout positive 
termination rate (in a 
period). 

Number of dropouts applying 
for services. 

Number of dropouts enrolled. 

Number of dropouts terminated. 

Repeat of above indicators for 
all youth. 

Average cost per positive 
outcome (in a period). 10/ 



Number of reorganization plans. 

Number of staff hired to 
implement policy. 

Allocation of bonus money 
among SDAs and subcontractors. 

List of all prograTi changes 
made to accommodate policy. 

Number cf changes in client pathway. 

Number of new local coordination 
agreements. 

PIC/staff reaction to policy. 

Number of dropouts enrolled 
and number with positive 
outcomes by racial/etnnic 
group (in a period) 



Evidence indicating dropout 
rates are increasing or 
decl ining. 

Program changes in response 
to rate fluctuations. 




10/ If Policy A (Exhibit 4) is implemented, states can effectively 
control these costs through the size of the bonus payments. There- 
fore, this objective has a good chance of being met if the state 
i*nplementation strategy is sound. 
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There are several points worth noting about these indicators. First, 
most of them are numeric, but some are verbal. In practice, numeric 
indicators will need to be supplemented by brief explanations of how 
local officials went about implementing the policy. This information can 
be collected through telephone conversations with local staff, a mail 
survey or other means. 11/ Second, indicators for the third objective 
are intended to capture the policy's influence on the mission, work, 
coordination and social components of local agencies. These indicators 
are especial ly useful in assessing the success of the policy's imple- 
mentation strategy. Third, althouyh the indicators refer to a single 
SDA, they can also be used to describe trends across SDAs. Thus, the 
above indicators can be collected for all SDAs and compared in some way, 
such as by computing averages or displaying the numbers in tables, charts 
and graphs. For example, the average youth dropout placement rate among 
all SDAs can be calculated for each month over a six month period and 
displayed in a graph. 

In the end, though, those responsible for monitoring the policy must 
interpret the numbers and decide whether the policy is working properly. 
This task is made easier if the criteria for determining whether a policy 
satisfies or fails to meet an objective is stated explicitly beforehand. 
For example, in the youth dropout problem, the effectiveness objective 
requires that 75 percent of the SDAs satisfy their performance standards 
for youth dropouts after policy implementation. Is the policy a success 
if only 70 percent achieve the objective? How about ^iQ percent? If 
multiple objectives exist, success or failure may be based only on 
indicators for the most important objective. If these issues are settled 
beforehand, officials will be in a better position to int^srpret findings 
during the implementation stage. 

If the policy is not working correctly, officials mujt use the available 
evidence (or collect additional information) to decide whether the under- 
lying theory or implementation strategy (or both) are at foult. 
Normally, the decision is to discard the policy whenever the underlying 
theory proves to be faulty. [1,9] But if the theory is sound but imple^ 
mentation has failed, the policy can often be corrected to achieve 
intended results. 

In closing, a final issue must be raised: who should condi*'"'*' the 
monitoring? Ideally, if a policy-making forum (consisting of stjlc, SDA 
and subcontractor representatives) developed tne policy, it st.uld also 
be responsible for monitoring it. The reason for this is simplr: not 
only , is this group probably the most knowledgeable about the pch'cy and 
the problem(s) it addresses, it also is likely to have the support of 
local officials. Thus, while its attempts to monitor policy at the local 
level may not be welcomed with open arms, they will likely face the least 
resistance. If monitoring is performed solely by the state, local actors 
might view it as part of the state's compliance activity. The net result 
here might be lack of cooperation by local actors or even the provision 
of inaccurate (but favorable) information about pol icy outcomes. 
However, if state monitoring is endorsed by the policy-making forum, it 
too might gain local acceptance. 




11 / The Grinker-Wal ker and Westat studies contain survey instruments 
that could be adapted for this purpose. [3,14] 
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CONCLUSION 



The purpose of this volume has been to describe a general approach for 
studying how state policies influence local operations. Although the 
focus of the illustrations has been youth policy issues, the approach can 
be adapted to most problems requiring a policy solution at the state 
level. The approach is based on the simple proposition that policy 
issues in JTPA can affect all levels of the state service delivery 
system, and that state, SDA and subcontractor officials must work 
together to solve their common problems. Thus, while states, SDAs and 
subcontractors were created to perform different functions in JTPA, the 
policy-making process is the coordination tool for bringing them together 
when problems arise. 
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APPENDIX A 

THE STATEWIDE JTPA SERVBCE DELSVERY SYSTEM 



Structure of the System 
Coordination and Accountabiiity 
State Performance Controi Systems 
SDA Performance Controi Systems 
Pros and Cons of Performance Control Systems 
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Understanding how JTPA works begins at the state level with a system-wide 
view of the prografii. This perspective is important because SDAs are not 
islands; they are part of a multi-organization system coordinated at the 
state level as defined in the Act. What goes on in any one organization 
cannot be understood fully without also looking at its relations with 
other organizations in the system. In this appendi.x a model of the 
statewide JTPA service delivery system is presented. A discussion of 
state-SDA relations within the model follows. 

A word of caution is needed before we begin. Parts of this section have 
an air of finality, as if the model perfectly matches what goes on in the 
JTPA service delivery system. That is not true, because all models— 
whether they are of organizations, airplanes or persons— are a 
simplification of reality. Real organizations are enormously complex. 
Models help us manage the complexity, to simplify it and sharpen its 
featur making it easier to understand. 

The models presented in this section may not exactly match your own 
program or image of JTPA. But, they should nonetheless serve as building 
blocks for developing and implementing process evaluations customized to 
your own program. 



STRUCTURE OF THE SYSTEM 

For most states, the JTPA service delivery system consists of three major 
organizations, the state, service delivery areas and subcontractors. The 
numbers of each organization vary considerably across states. In eleven 
states no SDAs exist; the state provides all services to parti- 
cipants. [1] SDAs may choose either to provide all services or sub- 
contract all or part of this function to other organizations in the local 
area. SDAs providing most services to participants may also elect to 
establish satellite offices to improve access. In some cases, sub- 
t;ontractors may be a state agency. Due to the coordination mandate, SDAs 
vand their subcontractors) are often linked with organizations outside 
the system, most typically local welfare offices. [2] To a great degree. 
Congress has established the roles and responsibilities of each 
organization in the model, which may be summarized briefly as 
follows [3,4]: 



STATES (GOVERNORS) 

Establish the State Job Training Coordinating Council (SJTCC). 

Designate Service Delivery Areas (SDAs), approve locally developed 
plans, and distribute grant funds to localities based on formulas 
established in the act. 

Monitor local program performance, prescribe variations in 
performance standards based on special conditions in the state, and 
award incentive bonuses for exceeding goals (or take action, in- 
cluding sanctions, when performance fails to meet standards or 
remains poor). 
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Establish and administer a new statewide dislocated worker program, 
a discretionary older worker program, a coordination and special 
service program, and a state labor market information system. 
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SERVICE DELIVERY AREAS 
Establish Private Industry Councils. 

• Develop Job Training Plans. 

• Implement Title II-A and Title III (where applicable) programs as 
stated in their Job Training Plans in accordance with performance 
standards and coordination criteria established by the state. 

• Monitor program performance of subcontractors (if applicable). 

SUBCONTRACTORS 

• Develop and submit proposals for training programs contained in SDA 
Job Training Plans or state training programs. 

• Implement training programs in accordance with performance standards 
established by the state or SDA and specified in legal subcontract 
agreements. 

By the model, most services are delivered to participants at the 
subcontractor level in most states [3, 4, 5]; what "goes on" at this 
organizational level essentially determines the services, or treatments, 
that participants receive in the system. [6] 

Characteristics of the JTPA system that pose challenges to administration 
and evaluation alike are as follows: 

• Multiple organizations: the state, local service delivery areas and 
subcontractors form a statewide system of employment and training 
services. 

• Decentralization: Several organizations (state, SDA and 
subcontractors) are responsible for administering JTPA. 

• Local autonomy: Each organization is free to develop its own 
programs, creating nonstandard forms of treatment and 
implementation. 

• Dynamic programs: Treatment and implementation change over time. 

• Multiple delivery strategies: Many treatments are possible to 
achieve a set of outcomes. 

• Conflicting/multiple c^als: Multiple goals may exist, some possibly 
conflicting with each other. 

• Multiple levels of coordination: Administrative and/or service 
coordination may occur among subcontractors, between subcontractors 
and SDAs, and between SDAs and the state. 
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In short, JTPA is a highly complex service delivery system. The more 
SDAs there are in a state, and the more these SDAs subcontract services 
the more complex the system becomes. This raises an important question* 
If responsibility is divided across so ..,any organizations, how is 
accountability i n the system maintained? How is service delivery co- 
ordinated between the state and SDAs? Between SDAs and subcontractors? 
These questions are addressed in the remainder of ^_iiis appendix 



COORDINATION AND ACCOUNTABILITY 

Basically, states with multiple SDAs and subcontractors coordinate 
service delivery and maintain accountability through two mechanisms, 
standardization of outputs and performance control systems. [7,8] 

All service delivery systems are created for a purpose, to produce a set 
of outputs. Congress created JTPA to train or prepare economically 
disaovantaged youth and adults for employment. The Secretary 
.standardized these outputs by establishing performance standards for all 
states and SDAs. Allowed services could be delivered as each provider 
saw fit. Hence, subcontractors are not told how to run their training 
programs, only to meet performance standards of their subcontract 
Similarly, SDAs coordinate with the state in terms of their own perfor- 
mance standards. SDAs are expected to meet performance standards on an 
annual basis; how they do this is their own business. 

Outputs are standardized within a performance control system to achieve 
coordination and accountability throughout the statewide system 
Performance control systems consist of two parts, the plan and control 
The plan specifies the desired outputs, or set of performance standard^ 
at some future time. In JTPA, these plans include the Governor's co- 
ordination and special services plan, the job training plans of SDAs, and 
(performance) contracts of subcorvcractors. The purpose of control is to 
determine whether the standard ' has been achieved. Achievements are 
monitored through Management Information Systems (MIS) After 
performance standards are established for the SDA, its actual performance 
IS later measured in terms of these standards and the results fed back up 
to the state. Similar MIS- feedback occurs between SDAs and sub- 
contractors (or SDA satellite offices). 

Thus, performance control systems are concerned with overall -esults for 
given periods of time—not with specific actions or decisions. States 
use performance control systems to verify that SDAs meet thei: respective 
performance standards— but not to monitor SDA decisions and program 
activity. For SDAs that subcontract or have satellite offices, the major 
concern is that the subcontractor/office fulfills minimum contract 
requirements, that it contributes to overall SDA performance without 
squandering its resources. Together, state and '.DA performance control 
systems form s "chain of accountability" across organizations and, hence, 
in the system as a whole. 

In virtually all performance control systems (including JTPA), SDAs and 
service providers generally have considerable freedom to meet their 
respective standards. Why is this autonomy important? Mainly because 
SDAs, compared to the state, are closer to the local labor market and, at 
least in principle, are in a better position to develop programs serving 
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local eligibles and employers. Similarly, subcontractors, compared to 
the SDA, are directly involved in service delivery and, hence, are in a 
better position to design customized programs meeting the needs of their 
respective employers and participants. In principle, the more services 
are designed to meet local participant and employer needs, the more 
effective the services should be. And the more effective the service, 
the more likely that performance standards will be met. 

Another reason for SDA and subcontractor autonomy is the design of the 
service delivery system itself. In JTPA states, SDAs and subcontractors 
are independent organizations joined together through loose adminis- 
trative agreements. Each organization lacks the authority— as well as 
the administrative resources—to control most actions and decision-making 
in the oth r organizations. 

A final reason for service provider autonomy are limitations of the MIS. 
In general, a MIS may be used properly to monitor SDA or subcontractor 
performance. However, an MIS does not contain sufficient information for 
administrative organizations (that is, the state or an SDA that sub- 
contracts and/or has satellite offices) to make decisions for service 
providers. Many service decisions are based on "soft" information not 
contained in an MIS. Knowing that a national manufacturer plans to open 
a new plant in your community may have a greater influence on decision- 
making than all the information in a MIS. Further, a MIS oft;i reports 
aggregated, historic information which may be of little use for solving 
current and/or future management problems. Last, most administrative 
organizations lack the time to absorb all the information a MIS can 
offer. So, the wise administrative organization knows what it can*t know 
and lets the service providers stay independent. 

Performance control systems work when everything runs well. But what 
happens when performance standards are not met? This issue is addressed 
in the following sections. 



STATE PERFORMANCE CONTROL SYSTEMS 

States have distinct responsibilities in the performance control system. 
Some of their responsibilities are defined in the law, but others are the 
result of their position in the JTPA service delivery system. These 
responsibilities include the following: 

• Designs the (state) performance control system (including MIS); 

• Prepares an annual statement and coordination and special services 
plan; 

• Calculates performance standards for SDAs, -granting variations ^nd 
waivers to the standards within guidelines established by the 
Secretary; 

• Allocates Title II resources to SDAs in ac:ordance with formulas for 
meeting performance standards; 

• Provides certain support services, such as labor market information, 
to assist SDAs meet their performance standards; 
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• Monitors SDA performance; and, 

• Maintains limited personal contact with SDA. 

The main purpose of monitoring and personal contacts is to flag problems 
in the SDA. The state uses varying amounts of personal contact with SDA 
staff to supplement its MI5-based performance control system. Typically 
the information in the MIS cannot capture all the problems that an SDA 
might be experiencing. For example, an MIS usually cannot determine 
whether an SDA's problem is due to either its own poor performance or a 
downturn in the local economy. Further, SDAs may manipulate the MIS to 
hide Its problems from the state. Because of these limitations of the 
MIS states often keep in touch with SDAs to learn about their operations 
^^•^^ fnA^^^ toward foreseeing problems. But too much contact interferes 
with SDA autonomy. Thus, states walk a tightrope between being ignorant 
of SDA problems and becoming so familiar that they are tempted to 
interfere. ' 

If problems are detected, the state must first determine their source, 
either outside (such as economic dowr.ts.'rn in the local economy) or inside 
the SDA. If the evidence indicates that the problems are from outside 
the SDA, thero is little the state can do. Performance standards or 
resource allocations may be adjusted to reflect the new economic 
conditions, or perhaps the state might target an economic development 
program into the area. States face real constraints here because 
they— as well as the SDA--do not control the environment in which the SDA 
operates. So, when the cause of poor performance is in the environment 
and beyond the control of the SDA, the SDA has really only one choice: 
adapt to the change as best it can and wait until better times arrive. 

But, what if the local economy is sound and the SDA itself seems to be 
the source of the problem? States have basic options to choose from 
most defined in Sections 105 and 105 of the JTPA. If the probleni(s) are 
neither too severe nor urgent, the state (i.e., the Governor) may use the 
state review process for job training plans to correct the problem. For 
example, in the SDA's next plan submission, the state may require a 
separate report on the current status of the problem, the SDA's success 
in solving the problem, and whether future actions are planned or 
necessary. In short, the state may elect to continue periodic monitoring 
of the SDA problem, making sure th=it it is resolved, at best, or at least 
does not become any worse. 

On the other hand, if the problem 5 ',zth severe and urgent, the state 
may impose a reorganization plan on the SDA. This plan may include one 
or all of the following measures (Sec. 105): 

• Restructure the SDA's PIC; 

Prohibit the SDA from further use of designated subcontractors; 

• Select a different entity to administer the area; and, 

• Make any other changes the state deems necessary to improve 
performance. 
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The top three items in the list are ways the state may change the 
governance/management of the area. These are common^ potentially 
successful methods of solving (severe) problems in public and private 
organizations using performance control systems. [7 8] The last item 
really indicates that states have tremendous latitude and power in how 
they deal with SDA performance problems. But whether a state actually 
uses this power may depend greatly on the severity of the problem and the 
political context in which the problem exists. For example, a Governor 
may be very reluctant to make major changes in a SDA if its local elected 
officials have a history of supporting the Governor on various state and 
local issues. 

It may not be too surprising, then, that most states choose to provide 
some form of '-technical assistance" to the SDA having performance 
problems. This is a useful, potentially effective way of handling the 
performance problem. But the form of the technical assistance is 
critical for thif intervention to be successful. Typically, state 
technical assistance has been a management strategy, such as setti ng 
clearer goals or increased monitoring. In other words, a common state 
reaction to poor performance 1:' to tighten the screws. Unfortunately, 
this often does more harm than good. Why is this so? 

First, training/service programs are fundamental to achieving good 
performance. When a state attempt ^ to improve performance by either 
adding to the administrative/manage: <!ot burden or controlling service 
delivery, SDA effort previously directed at service is now used to 
perform nonservice (compl iance)-related activities. The emphasis 
switches from serving participants to satisfying performance standards 
and state-imposed reporting requirements. In essence, an ends-means 
reversal occurs, and actual service delivery suffers as a consequence, 
likely reducing SDA performance even further. 

Second, providing services effectively to participants is complex work 
requiring competent staff. If SDA staff are incompetent, no set of state 
plans, rules or orders can make them more competent. Furthermore, these 
measures often prevent competent staff from doing their work effectively. 

In sum, state technical assistance that merely tightens the screws 
(through various management strategies) will likely not prove to be an 
effective mechanism for improving SDA performance. Instead, technical 
assistance directed at improving services seems more appropriate, mainly 
because services are the "means" of achieving program "ends" (performance 
standards). Services may be improved, for example, by increasing the 
capability of staff, such as by gradually replacing incompetent staff 
with more qualified individuals, increasing staff skills (through staff 
continuing education), or hiring a consultant to raise staff levels of 
performance in specific areas where weaknesses may exist. Other 
mechanisms, such as altering program content, structure or service 
deliverers, may also lead to service improvements. 



SDA PERFORMANCE CONTROL SYSTEMS 

Performance control systems are also present in SDAs which subcontract or 
operate satellite offices. In these systems the home office monitors the 
performance of subcontractors or branch offices. The main concerns of 
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the home office are that each subcontractor/branch satisfy its own 
s;,andards, and their combined efforts satisfy the standards for the SDA 
as a whole. SDA responsibilities in its performance control system 
differ from the state. Major ones are listed below: 

• Managing the "strategic portfolio," or the opening and closing of 
subcontracts and satellite offices to provide a specific mix of 
services to specific participants; 

• Allocating the financial resources of the SDA within the constraints 
imposed by the JTPA and state allocations; 

• Designing the performance control system; 12/ and, 

• Monitoring the performance and activity of subcontractors/ satellite 
offices on an informal basis. 

Generally, the SDA performance control syste.i. works much like the state 
system: subcontractors and satellite offices require autonomy to make 
their own decisions; monitoring occurs through the MIS, supplemented by 
informal contacts. Likewise, if a performance p ?Mem(s) is detected, 
the SDA must first determine if the source of the p.oblem is outside or 
inside the subcontractor/satellite office. If the subcontractor or 
office is the source of the problem, the SDA has these (and other) basic 
means of intervention: 

• Cancel or not renew the subcontract; 

• Replace the manager in the satellite office; and, 

• Provide technical assistance, increase compliance monitoring, or 
both. 

Again, if the last action is taken, technical assistance/compliance that 
merely "tightens the screws" will likely divert resources from service to 
satisfying the additional requirements (imposed by the SDA), in most 
cases reducing performance of the subcontractor or satellite office. 
Actions that increase staff skills (e.g., staff training) or otherwise 
improve service effectiveness will likely result in better performance 
levels. 



PROS AND CONS OF PERFORMANCE CONTROL SYSTEMS 

Because most states have SDAs, most operate one form or another of a 
performance control system. However, a PIC or SDA may choose between 
subcontracting or delivering services itself, which translates into 
operating either a multi-organization performance control system or an 
MIS for tracking its own perforfnarxa. While the cnoice may be dependent 
on staff skids, other factors also influence this decision. First, 
performance control systems work hest in stable and simple environments. 



12/ In many cases, SDAs may simply extend the state system down to the 
subcontractor/satellite level, sometimes in modified form. 
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For SDAs, this means having a fairly stable, predictable local economy 
that is simple to understand and monitor. These environments allow SDAs 
and their subcontractors to standardize their services and operating 
systems, creating stability in organizations. Managers often agree that 
satisfying performance standards is easier under these conditions. 

Complex and dynamic environments, on the other hand, are less supportive 
of performance control systems . For exampl e , devel opi ng performance 
standards for a rapidly changing local economy and "hard to serve" 
participants is at best a difficult task. Performance in such environ- 
ments is likely determined more by events happening outside the SDA than 
the relatively short-term services participants receive. 

Performance control systems also work best in highly diversified areas. 
For example, if an SDA is serving a large number of target groups, sub- 
contracting services by target group is one legitimate way of 
establishing a performance control system. Such diversification would 
make sense if each subcontractor provided a unique set of services 
customized to meet the specific employment and training needs of its 
target group. Other Vorms of diversifying, such as by region or service, 
are also possible. And once an SDA has decided to diversify (or sub- 
contract), it likely finds that adding/deleting subcontractors becomes 
relatively easy. However, if an SDA's services are the same for most 
participants across all regions, then it might make more sense for the 
SDA to provide most services itself. 

Performance control systems also have two major, negative 
characteristics. First, they tend to stifle innovation. Innovation (of 
services) involves risk— a risk that performance standards may not be met 
if the innovation fails. Because performance is the bottom line in most 
SDAs, PICs may be less willing to innovate, particularly when current 
services are meeting standards. Thus, performance control systems 
encourage conservative program management. 

Second, these systems emphasize economic consequences but downplay soc ial 
ones. Every major decision that a PIC or SDA director makes has economic 
as wel 1 as social consequences for most groups. The economic 
consequtnces of JTPA are its performance standards--percent employed, 
average wage rates and so forth. They are easily measured and, hence, 
are easily integrated in:o the MIS and performance control system. 
Hence, they often have the uriivided attention of the PIC, management and 
most everyone else in the system. 

But there are social consequences to decision-making as well. These 
consequences are often hanJ to measure and, hence, are rarely included in 
performance control sysU. ris. Examples include decisions to orient 
services around the job-ready instead of the hard-to-serve, to ignore 
youth below the age of 18 because their outcomes (or competencies) are 
difficult to measure, or to emphasize placements instead of the gain in 
skills or wages from training. When the screws are turned really tight 
and pressures mount, SDA (or subcontractor) management may have no other 
choice but to ignore social consequences of their decisions and to meet 
performance standards-^^no matter what. Thus, performance control systems 
direct energy and resources toward achieving results measured chiefly in 
economic terms and may, under the worst of circumstances, inhibit social 
responsiveness snd responsibility. 
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Is it possible to develop a performance control system that accounts for 
social consequences of decision-making under JTPA? Well, yes and no. 
Although we cannot measure the social dimension precisely in most 
instances, crude measures may be possible. Let's use "need" for serv5>'';'>!; 
as an example. On a scale of 0.00 to 1.00, an unemployed, single BOtn-- 
on welfare with children and no skills or work history might be giv- 
1.00'— having the greatest need. Someone who is unemployed 
unskilled might be givisn a ".50". Someone who is unemployed bul ^lao 
good job history might be given a ".25"— that is, having lowest naed. 
Then, placements could be adjusted, or "weighted," by these need factors, 
such that a welfare mother counts as a full-placement (1.0) an 
unemployed/unskilled person counts as a half-placement (.50), and the 
last counts as a quarter-plscement (.25). This would mean that, to 
achieve one placement, a provider could either place one welfare mother 
or two unemployed/unskilled persons or four skilled persons. This is 
just one way of including social and economic factors in a performance 
control system. However, such adjustments would require that reliable 
and valid measures for the concept of "need" be available, and that 
sufficient resources exist to treat the hard-to-serve. 



SUMMARY 

This appendix has shown how performance control systems work to maintain 
a chain of accountability" throughout the statewide JTPA service 
delivery system. Each SDA is concerned that the combined efforts of 
subcontractors satisfy its performance standards; states are concerned 
that SOAs meet their respective performance standards to satisfy state 
accountability requirements. Thus, although revenues are spread across 
multiple organizations at different levels in the system, accountability 
is maintained through the chain and, hence in the system as a whole. 
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APPENDIX B 
COST-EFFECTBVENESS ANALYSIS IN AN 
EMPLOYMENT AND TRAINING PROGRAM 
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THE COST EFFECTIVENESS TECHNIQUE 13/ 
By ~ 
HENRY M. LEVIN 



The use of cost-effectiveness analysis in evaluation can best be shown by 
constructing a simple illustration of its use. From this hypothetical 
example it is possible to grasp the ddoed dimension that the approach 
provides for social decision making. Following this illustration we will 
at'-.empt to describe the principles and procedures for considering and 
m-^asuring costs and effects. 

Assume that we are engaged in evaluating programs designed to reduce the 
rate of recidivism of convicts who are released from the state prison. 
For purposes of this vxercise let us define the rate of recidivism as 
that proportion of former prisoners who are arrested and convicted of 
criminal acts within five years of being released. The existing program 
is one that keeps a record of the addresses and employment circumstances 
of former prisoners, as well as requiring those who are released on 
parole to report periodically to their parole officers. Beyond these 
bookkeeping relationships there are no systematic attempts to provide 
either jobs or psychological counseling and assistance. 

To reduce an apparently high rate of recidivism, the State Prison 
Authority wishes to consider a number of alternative programs for ex- 
prisoners. These include: (1) a job placement program, (2) a psychol- 
ogical services program, and (3) a program that combines both job 
placement and psychological services. With the assistance of evaluation 
experts, a major social experiment is initiated to determine the impacts 
of the three alternatives on the rate of recidivism. For a period of six 
months all of the released male prisoners who are returning to the major 
cities of the state are assigned randomly to one of four groups: (1) job 
placement, (2) psychological services, (3) combination of job placement 
and psychological services, or (4) normal existing arrangements for 
ex-prisoners. 

After five and one-half years the experiment ends with the following 
results. The five-year rates of recidivism were 15 percent for those in 
the job placement program; 26 percent for those in the psychological 
services program: 12 percent for those in the combination program; and 
37 percent for those ex-prisoners who received no special treatment. 
Based upon this appraisal it appears that all programs were more 
successful than the existing approach, but the combination treatment 
showed the best results, followed by the job placement program and the 
psychological services one. Under normal circumstances the evaluation 
might have ended hev*e with the policy recommendation that the combination 
program be selected, but we wish to review the costs as well. 



13. Levin, Henry M. "Cost-effectiveness Analysis in Evaluation 
Research." M. Guttentag and E. L. Struening (eds). Handbook of 
Evaluation Research. Volume 2. Beverly Hills, CA; Sage. i:,75. 
Copyright 1975 by Sage Publications, Inc. Reprinted by permission 
of Sage Publications, Inc. 



EKLC 



1S7 



53 



Appendix Exhibit 1 shows a hypothetical cost-effectiveness comparison of 
the anti-recidivism programs for released prisoners. The three 
experimental treatments discussed above are compared with the results of 
the normal program. For simplicity we have assumed that exactly equal 
numbers-- 10, 000 subjects--were assigned to each group. The five-year 
recidivism rates are shown, and beneath them in parentheses are noted the 
rankings of the result (I is best). Based upon these rates we can 
calculate the number of persons who were not recidivous, who were not 
arrested and convicted of criminal acts within five years of their 
release from state prison. 

Hypothetical total costs for each program are shown on the next line. It 
is assumed that the normal program is least expensive because it is 
essentially an "auditing" or "accounting" approach for maintaining infor- 
mation on the location and activities of each ex-prisoner. The joc 
placement and p:iychological services programs are most costly, and the 
combination of them is the most expensive. It is assumed that some 
aspects of the combination program are duplicated in the separate ones, 
so that the cost of the combination approach is somewhat less than the 
combined tctal of the two separate components. When these costs are 
divided among the number of subjects in each group, it is clear that the 
"normal program" is the least expensive and the combination shows the 
highest average cost per subject. 

But the average cost per subject tells us nothing about the cost for 
obtaining the desired criterion, namely the reduction in recidivous 
subjects. The next line compares the average cost per nonrecidivous 
subject. According to this comparison, the "normal" program shows the 
lowest average cost per nonrecidivous subject, followed by the job 
placement program and the psychological services one, while the 
V nbination program seems to have the highest cost. But this comparison 
is not completely valid because the programs are being credited for 
subjects who probably would not have been recidivous even in the absence 
of the programs. For example, the "normal program" is merely a book- 
keeping effort. While maintaining regular contact and information on the 
activities and whereabouts of released prisoners will have some effect on 
the likelihood of their returning to crime, it is likely that even in the 
absence of such a program a significant portion of the men would not be 
recidivous. Accordingly, using the total number of nonrecidivous 
subjects as a basis for calculations understates the cost of each 
"success", because it counts any nonrecidivous person as a credit to the 
program. This bias is most severe for the "normal program," but it is 
also evident for the other groups. 
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APPENDIX EXHIBIT 1 



COST-EFFECTIVENESS COMPARISON OF 
ANTI-RECIDIVISM PROGRAMS FOR RELEASF ■ PRISONERS 



. Treatment 




Job 
Placement 


Psychological 
Services 


Combination 
of These 


Normal 
Program 


Experimental 
Population 


10,000 


10,000 


10,000 


10,000 



Five-year rate 
of recidivism 



Number of persons 
not recidivous 



Total Cost 

Average cost 
per subject 



Average cost per 
nonrecidivous subject 



Number of persons not 
recidivous in 
comparison with 
normal program 

Additional cost 
beyond normal program 



Marginal cost per 
additional non- 
recidivous subject 



15 

(2) 



8,500 
(2) 



26 
(3) 



7,400 
(3) 



,12 
(1) 



8,800 
(1) 



37 
(4) 



6,300 
(4) 



$10,000,000 $ 9,000,000 $16,000,000 $ 5,000,000 



$1,000 
(3) 



$1,176 
(2) 



2,200 
(2) 



$2,273 
(1) 



$ 900 
(2) 



$1,216 
(3) 



1,100 
(3) 



$1,600 
(4) 



$1,818 
(4) 



2,500 
(1) 



$5,000,000 $4,000,000 $11,000,000 
(2) (1) (3) 



$3,636 
(2) 



$ 500 
(1) 



$ 794 
(1) 



$4,400 
(3) 



Source: Levin (1975). Table 5.1. 
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These preliminary calculations lead us to the final set of results in 
Appendix Exhibit 1. If we assume that the "normal program" is required 
by law, then we do not have the opportunity to eliminate it. Assuming 
that each of the other treatments also provides for regular contact with 
the person and collection of the relevant information, the policy 
question is what cost (for each additional nonrecidi vous person) is saved 
by one of the special treatments. In comparison with the "normal 
program," the job placement, psychological services, and combination 
programs enable an additional 2,200, 1,100, and 2,500 persons, re- 
spectively (out of 10,000 subjects), to avoid returning to crime and 
prison. As we noted from the recidivism rates, the combination program 
seems most successful , followed by job placement and psychological 
service programs. 

But the additional costs beyond the normal program vary from treatment to 
treatment. While the job placement program costs an additional $5 
million, the psychological services cost another $4 million, and the 
combination progr^im had an added cost of $11 million in comparison with 
the standard approach. From these data we can calculate the marginal or 
additional cost for each additional nonrecidi vous subject. This varies 
from $2,273 for tho job placement program to $4,400 for the combination 
approach. In other words, it cost about half as much to reduce 
recidivism by one person via job placement as via the combination 
program. The psychological services program was about midway in cost per 
additional nonrecidi vous subject. 

In summary, although the experiment demonstrated that the combination 
program showed the most success in reducing recidivism, its higher cost 
would not justified. Rather, job placement would appear to be the 
most promising approach from the cost-effectiveness vantage point. 

To illustrate the impact of choice of approach on the budget of the 
social agency that is administering the program, we can calculate that 
the cost of "saving" an additional 1,000 released prisoners from 
returning to prison is about $2.3 million under the job placement 
program, $3.6 million under the psychological services approach, and 
$4.4 million under the combination program. 

This illustration brings out a number of points. First, often what 
appears to be the most "effective" program may not be the most >:^t- 
effective . In this instance the governmental cost for each addition"aT 
nonrecidi vous man would have been twice as high had the most "effective" 
treatment been used as a basis for program selection. Second, not only 
may effectiveness rankings differ from cost-effectiveness rankings, but 
the total social costs of making the wrong choice by not considering 
program costs may be substantial. Third, different measures of cost may 
provide different implications, as a comparison of the figures on average 
cost per subject, average cost per nonrecidi vous subject, and marginal 
cost per additional nonrecidi vous subject show. Therefore, it is 
imperative that the appropriate cost comparison be used in order to 
obtain appropriate results. 
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